-- Logs begin at Fri 2022-09-30 08:51:32 UTC, end at Fri 2022-09-30 10:08:21 UTC. -- Sep 30 09:25:44.573794 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 30 09:25:44.581302 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 30 09:25:44.584839 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: *** Starting uWSGI 2.0.18-debian (64bit) on [Fri Sep 30 09:25:44 2022] *** Sep 30 09:25:44.584839 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 30 09:25:44.584839 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 30 09:25:44.584839 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: nodename: dv-cinder-master-3-iscsi Sep 30 09:25:44.584839 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: machine: x86_64 Sep 30 09:25:44.584839 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: clock source: unix Sep 30 09:25:44.584839 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: pcre jit disabled Sep 30 09:25:44.584839 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: detected number of CPU cores: 4 Sep 30 09:25:44.584839 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: current working directory: / Sep 30 09:25:44.584839 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: detected binary path: /usr/bin/uwsgi-core Sep 30 09:25:44.584839 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: your processes number limit is 47680 Sep 30 09:25:44.584839 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: your memory page size is 4096 bytes Sep 30 09:25:44.584839 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: detected max file descriptor number: 2048 Sep 30 09:25:44.584839 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: lock engine: pthread robust mutexes Sep 30 09:25:44.584839 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: thunder lock: enabled Sep 30 09:25:44.585643 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 30 09:25:44.585643 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 30 09:25:44.625610 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: Python main interpreter initialized at 0x55bdbeb3f9c0 Sep 30 09:25:44.625610 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: python threads support enabled Sep 30 09:25:44.625610 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: your server socket listen backlog is limited to 100 connections Sep 30 09:25:44.625610 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: your mercy for graceful operations on workers is 90 seconds Sep 30 09:25:44.625965 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: mapped 403077 bytes (393 KB) for 2 cores Sep 30 09:25:44.626043 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: *** Operational MODE: preforking *** Sep 30 09:25:44.626297 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: *** uWSGI is running in multiple interpreter mode *** Sep 30 09:25:44.626297 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: spawned uWSGI master process (pid: 81676) Sep 30 09:25:44.626500 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 30 09:25:44.627699 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: spawned uWSGI worker 1 (pid: 81684, cores: 1) Sep 30 09:25:44.627699 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: spawned uWSGI worker 2 (pid: 81685, cores: 1) Sep 30 09:25:44.627699 dv-cinder-master-3-iscsi devstack@keystone.service[81676]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 30 09:25:48.950923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 30 09:25:48.950923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: from cryptography.utils import int_from_bytes Sep 30 09:25:48.951722 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 30 09:25:48.951722 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: from cryptography.utils import int_from_bytes Sep 30 09:25:48.957680 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 30 09:25:48.957680 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: from cryptography.utils import int_from_bytes Sep 30 09:25:48.958149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 30 09:25:48.958149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: from cryptography.utils import int_from_bytes Sep 30 09:25:49.005213 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 30 09:25:49.005392 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 30 09:25:49.005584 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] command line args: [] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 30 09:25:49.005869 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 30 09:25:49.006049 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ================================================================================ {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 30 09:25:49.006301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] admin_token = **** {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.006547 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] config_dir = [] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.006730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 30 09:25:49.006814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.006928 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] config_source = [] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.007015 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 30 09:25:49.007090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] debug = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.007161 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] command line args: [] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 30 09:25:49.007291 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.007790 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 30 09:25:49.007867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ================================================================================ {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 30 09:25:49.007928 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.008071 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] admin_token = **** {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.008149 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.008308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] config_dir = [] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.008422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.008532 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.008641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.008739 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] config_source = [] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.008840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] list_limit = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.008942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] debug = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.009050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] log_config_append = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.009155 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.009283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.009381 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.009493 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] log_dir = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.009597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.009704 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] log_file = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.009704 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.009915 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.009998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.010120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.010198 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.010324 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] list_limit = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.010324 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] log_config_append = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.010510 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.010607 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.010684 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.010818 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] log_dir = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.010901 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.010980 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] log_file = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.011096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.011224 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.011316 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.011437 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.011518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.011639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.011721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.011823 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.011937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.012024 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.012169 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.012291 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.012412 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.012490 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.012656 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.012742 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.012892 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.012969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.013128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.013362 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.013494 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.39/identity {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.013573 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] publish_errors = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.013645 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.013791 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.013879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.014020 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.014103 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.014180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.014340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.014421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.014669 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.014862 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.39/identity {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.014952 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] standard_threads = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.015031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] publish_errors = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.015152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.015269 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.015354 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.015500 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.015578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.015656 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] use_journal = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.015734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.015832 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] use_json = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.015917 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.016005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] use_stderr = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.016129 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.016221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] use_syslog = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.016376 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] standard_threads = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.016462 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.016674 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.016751 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.016912 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.017045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.017128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.017250 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.017331 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] use_journal = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.017447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.017530 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] use_json = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.017618 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] use_stderr = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.017696 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.017792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] use_syslog = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.017871 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.018026 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 30 09:25:49.018185 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.018288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.018373 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.018532 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.018627 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.018825 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.018914 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.019074 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.019159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.019310 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.019394 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.019553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.019640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.019787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.019871 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.020030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.020160 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.020294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.020427 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.020564 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.020690 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.020825 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.020952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.021080 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.021219 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.021345 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.021468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.021592 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.021717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.021842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.021975 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.022100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.022224 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.022363 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.022488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.022647 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.022786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.022924 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.023031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.023097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.023156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.023220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.023371 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.023533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.023659 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.023785 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.023919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.024044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.024244 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.024373 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.024496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.024620 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.024748 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.024874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.025009 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.025134 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.025257 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.025388 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.025518 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.025651 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.025785 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.025911 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.026039 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.026183 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.026263 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.026344 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.026423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.026495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.026571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.026737 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.026823 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.026905 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.026905 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.027041 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.027144 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.027240 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.027336 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.027440 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.027543 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.027640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.027735 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.027831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.027959 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.028070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.028178 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.028255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.028332 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.028404 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.028525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.028610 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.028684 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.028805 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.028901 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.029029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.029179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.029308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.029470 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.029551 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.029707 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.029793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.029940 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.030025 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.030169 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.030250 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.030414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.030493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.030668 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.033675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.033792 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.033873 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.033984 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.034091 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.034177 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.034825 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.034912 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.034995 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.035069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.035140 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.035210 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.035281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.035357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.035430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.035502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.035573 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.035649 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.035722 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.035800 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.035874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.035945 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.036024 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.036097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.036097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.036226 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.036352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.036431 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.036503 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.036574 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.036644 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.036721 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.036801 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.036881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.036953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.037023 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.037094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.037164 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.037313 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.037388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.037460 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.037536 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.037609 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.037681 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.037764 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.037838 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.037914 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.037914 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.038105 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.038105 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.038257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.038338 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.038411 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.038483 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.038554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.038646 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.038720 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.038720 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.038869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.038948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.039020 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.039092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.039163 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.039241 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.039315 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.039386 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.039457 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.039529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.039654 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.039740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.039820 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.039893 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.039964 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.040035 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.040095 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.040173 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.040246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.040410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.040490 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.040643 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.040728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.040927 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.041011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.041163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.041245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.041416 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.041495 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.041650 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.041731 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.041887 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.041971 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.042115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] auth.external = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.042202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.042344 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.042424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.042576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.042673 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.042842 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.042924 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.043084 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.043192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] auth.password = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.043310 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] auth.token = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.043397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.043543 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.043624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.043775 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.043857 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.044019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.044100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.044246 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.044325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.044488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.044568 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] auth.external = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.044730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.044811 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.044955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.045033 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.045225 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] credential.caching = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.045314 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.045473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.045601 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] auth.password = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.045717 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.045840 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] auth.token = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.045963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.046085 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.046203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.046324 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.046442 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.046576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.046703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.046834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.046917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.047086 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.047168 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.047337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.047417 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.047573 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.047655 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.047814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] credential.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.047899 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.048065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.048145 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.048341 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.048430 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.048569 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.048655 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.048824 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.048903 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.049059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] federation.caching = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.049140 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.049291 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.049370 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.049537 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.049616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.049764 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.049962 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.050571 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.050718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.050840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.050922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.051169 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.051338 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.051414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.051476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.051642 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.051723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.051894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.051984 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.052132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.052213 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.052376 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.052455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] federation.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.052611 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] identity.caching = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.052693 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.052857 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.052937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.053102 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.053183 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.053337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.053419 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.053569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.053649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.053814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.053902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.054045 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.054128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.054279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.054359 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.054525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.054659 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.054799 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.054882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.055052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.055132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.055293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.055373 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.055549 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.055631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.055798 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.055877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.056035 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.056122 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.056276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.056357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.056526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.056605 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.056758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.056845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.057001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.057080 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.057249 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.057328 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.057484 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.057579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.057713 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.057793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.057970 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.058089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.058202 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.058296 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.058436 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.058550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.058694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.058831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.058939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.059088 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.059179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.059349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.059430 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.059604 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.059684 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.059846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.059926 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.060078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.060157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.060320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.060399 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.060562 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.060641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.060787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.060865 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.061021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.061100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.061247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.061328 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.061480 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.061560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.061712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.061791 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.061950 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.062038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.062183 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.062263 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.062431 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.062528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.062749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.062931 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.063003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.063102 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.063246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.063339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.063496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.063576 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.063733 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.063811 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.063975 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.064054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.064211 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.064291 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.064453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.064531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.064688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.064766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.064919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.user = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.064999 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.065157 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.065236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.065394 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.065483 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.065671 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.065761 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.065916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.065999 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.066148 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.066225 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.066382 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.066503 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.066641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.066787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.066920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.067052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.067200 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.067330 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.067456 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.067580 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.067705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.067831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.067965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.068092 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.068225 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.068395 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.068533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.068617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.068744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.068821 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.068894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.068966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.069038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.069118 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.069193 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.069354 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.069435 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.069596 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.069676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.069855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.069939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.070098 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.070178 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.070338 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.070418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.070580 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.070676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.070846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.070925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.071080 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.071159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.073825 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.073909 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.073971 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.074031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.074090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.074149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.074217 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.074279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.074339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.074398 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.074457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.074516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.074582 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.074685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.074749 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.074808 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.074867 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.074926 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.075063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.075127 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.075187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] resource.caching = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.075245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.075304 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.075363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.075429 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.075491 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.075549 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.075608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.075672 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.075732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.075798 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.075860 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.075918 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.075990 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.076050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.076109 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.076174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.076236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.076295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.076353 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.076412 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] role.caching = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.076471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.076561 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] role.driver = sql {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.076626 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.076686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.076745 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.076804 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.076868 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] resource.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.076936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.077006 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.077066 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.077211 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.077302 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.077402 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.077541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.077655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.077786 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.077910 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.078032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.078156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.078346 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.078488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.078635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.078718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.078811 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.078884 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.078969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.079117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] role.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.079197 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.079372 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] role.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.079452 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.079675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.079904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.079989 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.080143 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.080224 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.080391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.080481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.080639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.080719 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.080885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.080964 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.081156 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.081244 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.081389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.081470 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.081641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.081720 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.081880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.081968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.082115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.082194 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.082344 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.082463 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.082581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.082705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.082813 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.082894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.083089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.083243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.083326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.083484 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.083566 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.083738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.083818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.083983 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.084062 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.084211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.084293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.084444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] token.caching = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.084529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.084677 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.084757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.084921 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.085003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.085153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.085233 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.085445 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.085535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.085692 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.085774 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.085926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.086006 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.086168 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.086248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.086409 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.086491 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.086678 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.086767 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.086914 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.087007 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.087150 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.087230 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.087383 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.087462 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.087626 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.087715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.087864 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] token.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.087943 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.088108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.088188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.088347 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.088435 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.088589 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.088677 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.088849 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.088929 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] database.connection = **** {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.089085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.089165 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.089329 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.089416 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.089565 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.089644 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.089801 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.089882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.090035 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.090123 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.090268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.090347 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.090500 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.090579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.090755 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.090849 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.090997 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.091077 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.091230 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.091310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.091528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.091622 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.091816 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.091919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.091997 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.092071 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.092217 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.092299 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.connection = **** {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.092450 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.092528 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.092673 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.092754 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.092909 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.092989 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.093141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.093223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.093297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81685) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 30 09:25:49.093453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.093695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.093935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.094176 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.094426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.094691 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.094943 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.095186 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.095440 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.095682 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.095931 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.096166 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.096410 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.097172 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 30 09:25:49.097358 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 30 09:25:49.123495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.123723 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.123945 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.126413 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.126675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.126910 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.132416 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.135809 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.164165 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.164389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.165926 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.166158 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.226132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.226540 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.275442 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.275673 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.276476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.276717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.441255 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.441479 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.441702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.441893 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.441982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.442123 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.442207 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.442337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.442423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.442556 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.442801 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.443021 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.465823 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.466050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.466274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.466500 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.466769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.466986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.467203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.467967 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.468207 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.468440 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.468669 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.468892 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.469110 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.469328 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.478877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.480435 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.497943 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.498055 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.498250 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.498341 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.498475 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.498560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.498718 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.498805 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.498958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.499083 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.499196 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.499290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.499410 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.499492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81685) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 30 09:25:49.522676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.522784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.522875 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 30 09:25:49.522984 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 30 09:25:49.523211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.523329 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.523411 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 30 09:25:49.523532 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 30 09:25:49.523673 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.523813 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.523937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.524084 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.524197 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.524352 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.524457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.524616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.534122 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.534962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.535045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.535236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.535311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.535535 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.535615 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.536701 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.542684 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81685) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.543246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.545569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81685) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.545853 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.546136 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.546232 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.546418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.546502 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.546728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.546815 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.546993 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.547090 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.547264 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.547357 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.547535 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.547618 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.547804 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.547888 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.548079 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.548162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.548347 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.548437 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.548625 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.548947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.571221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.572529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.572693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81685) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.572974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.574027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.574326 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.577644 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81685) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.577928 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.578210 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.579144 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.579434 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.579721 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.585135 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81685) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.585420 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.586896 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.587189 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.591066 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81685) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.591360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.591649 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.591939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.592214 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.592487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.592755 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.592906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.593029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.593208 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.593504 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.593819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.594106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.594520 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.594829 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.595136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.609334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.609609 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.611672 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.611964 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.613529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81685) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.615837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.616789 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.617084 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.619087 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.619499 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.621349 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81685) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.623703 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.624500 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81685) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.627056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.627777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.628101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.628387 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.628674 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.628964 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.629259 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.630342 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.630712 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.630988 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.631282 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.636378 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.636378 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.646089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81685) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.646377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.646670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.646934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.649009 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.649414 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.649688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.649959 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.653810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.655508 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.656939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.657029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.657217 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.658524 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.660539 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81685) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.660634 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.660824 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.660909 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.661114 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.661394 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.661778 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.663739 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.664030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.664327 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.664618 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.665014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.673003 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81685) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.673288 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.673574 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.673855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.674268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.674565 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.674897 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.676452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.679363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.679363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.679363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.679363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.679363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.679363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.691067 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81685) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.694012 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81685) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.694904 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.697008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.697849 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.698475 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.699982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81685) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.700246 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.700648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.700783 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.702312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.703811 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.704100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.704508 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.707691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.709137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81685) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.711533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.712045 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.712437 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.712725 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.713002 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.713087 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.715946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.716349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.719389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.719389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.720928 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81685) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.721231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.721512 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.724981 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.725342 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.725631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.728574 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81685) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.728859 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.729187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.729419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.729689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.729958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.730240 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.730527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.730857 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.731136 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.731407 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.731683 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.731949 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.732224 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81685) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.732810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 30 09:25:49.733099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.733386 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.733672 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.733952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.734231 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.734508 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.734826 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.735105 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.735379 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.735665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.735956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.736253 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.736608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 30 09:25:49.767175 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55bdbeb3f9c0 pid: 81685 (default app) Sep 30 09:25:49.771086 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1712d49-da85-4ec6-abff-fdd0559bc0be None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:25:49.771374 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1712d49-da85-4ec6-abff-fdd0559bc0be None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:25:49.771701 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1712d49-da85-4ec6-abff-fdd0559bc0be None None] PATH_INFO: `/v3` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:25:49.773034 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55bdbeb3f9c0 pid: 81684 (default app) Sep 30 09:25:49.775817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1/1] 10.222.0.39 () {50 vars in 791 bytes} [Fri Sep 30 09:25:49 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:25:57.281813 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3a7e39-9a59-4f8b-8303-20d259f18504 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:25:57.282315 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3a7e39-9a59-4f8b-8303-20d259f18504 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:25:57.282398 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3a7e39-9a59-4f8b-8303-20d259f18504 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:25:57.287205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1/2] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:25:57 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:25:57.292692 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69e370e-5cea-4bed-803b-620075cb3e1a None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:25:57.293136 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69e370e-5cea-4bed-803b-620075cb3e1a None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:25:57.293296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69e370e-5cea-4bed-803b-620075cb3e1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:25:57.365608 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: INFO dbcounter [None req-a69e370e-5cea-4bed-803b-620075cb3e1a None None] Registered counter for database keystone Sep 30 09:25:57.384985 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG oslo_db.sqlalchemy.engines [None req-a69e370e-5cea-4bed-803b-620075cb3e1a None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81685) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 30 09:25:57.391200 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writer thread running {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 30 09:25:57.569316 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG passlib.handlers.bcrypt [None req-a69e370e-5cea-4bed-803b-620075cb3e1a None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81685) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 30 09:25:57.569705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG passlib.handlers.bcrypt [None req-a69e370e-5cea-4bed-803b-620075cb3e1a None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81685) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 30 09:25:57.628283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-a69e370e-5cea-4bed-803b-620075cb3e1a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:25:57.629882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a69e370e-5cea-4bed-803b-620075cb3e1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:25:57.756827 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 2/3] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:25:57 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 465 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Sep 30 09:25:58.215664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3a7e39-9a59-4f8b-8303-20d259f18504 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:25:58.286039 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: INFO dbcounter [None req-5e3a7e39-9a59-4f8b-8303-20d259f18504 None None] Registered counter for database keystone Sep 30 09:25:58.307078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG oslo_db.sqlalchemy.engines [None req-5e3a7e39-9a59-4f8b-8303-20d259f18504 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81684) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 30 09:25:58.313134 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writer thread running {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 30 09:25:58.430777 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3a7e39-9a59-4f8b-8303-20d259f18504 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:25:58.435007 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac74285e-27f8-4bae-953b-b87b05fc9201 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:25:58.435625 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac74285e-27f8-4bae-953b-b87b05fc9201 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:25:58.435906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac74285e-27f8-4bae-953b-b87b05fc9201 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:25:58.436179 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac74285e-27f8-4bae-953b-b87b05fc9201 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:25:58.445075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac74285e-27f8-4bae-953b-b87b05fc9201 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:25:58.495448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: WARNING keystone.common.rbac_enforcer.enforcer [None req-ac74285e-27f8-4bae-953b-b87b05fc9201 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 30 09:25:58.495797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac74285e-27f8-4bae-953b-b87b05fc9201 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:25:58.505634 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 2/4] 10.222.0.39 () {58 vars in 1324 bytes} [Fri Sep 30 09:25:58 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 30 09:25:58.511883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69e370e-5cea-4bed-803b-620075cb3e1a None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:25:58.528716 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69e370e-5cea-4bed-803b-620075cb3e1a None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:25:58.533197 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836f6088-7bb4-4fbf-aff8-5bbd6440defb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:25:58.534304 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836f6088-7bb4-4fbf-aff8-5bbd6440defb admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:25:58.534606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836f6088-7bb4-4fbf-aff8-5bbd6440defb admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:25:58.534896 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836f6088-7bb4-4fbf-aff8-5bbd6440defb admin admin] PATH_INFO: `/v3/projects/f04f50d086c64df3bc18b4c8f8baadc4` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:25:58.536492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836f6088-7bb4-4fbf-aff8-5bbd6440defb admin admin] RBAC: Authorizing `identity:get_project(project_id=f04f50d086c64df3bc18b4c8f8baadc4)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:25:58.588800 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: WARNING keystone.common.rbac_enforcer.enforcer [None req-836f6088-7bb4-4fbf-aff8-5bbd6440defb admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 30 09:25:58.589155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836f6088-7bb4-4fbf-aff8-5bbd6440defb admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:25:58.593664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 3/5] 10.222.0.39 () {56 vars in 1207 bytes} [Fri Sep 30 09:25:58 2022] GET /identity/v3/projects/f04f50d086c64df3bc18b4c8f8baadc4 => generated 344 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:00.710449 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be62caa-a372-4608-af3e-7b9204b894f3 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:00.711027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be62caa-a372-4608-af3e-7b9204b894f3 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:00.711027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be62caa-a372-4608-af3e-7b9204b894f3 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:00.715153 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 3/6] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:00 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:00.720473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfacbf0-0ae8-4253-945b-1d5072a9e6c7 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:00.720866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfacbf0-0ae8-4253-945b-1d5072a9e6c7 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:00.721033 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfacbf0-0ae8-4253-945b-1d5072a9e6c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:00.741292 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-3dfacbf0-0ae8-4253-945b-1d5072a9e6c7 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:00.742863 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-3dfacbf0-0ae8-4253-945b-1d5072a9e6c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:00.750638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 4/7] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:00 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 30 09:26:01.019559 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be62caa-a372-4608-af3e-7b9204b894f3 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:01.031908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be62caa-a372-4608-af3e-7b9204b894f3 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:01.036641 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7ac155-857b-40bc-a1d1-7eeb7ef6cb0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:01.037248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7ac155-857b-40bc-a1d1-7eeb7ef6cb0e admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:01.037534 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7ac155-857b-40bc-a1d1-7eeb7ef6cb0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:01.037811 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7ac155-857b-40bc-a1d1-7eeb7ef6cb0e admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:01.045378 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7ac155-857b-40bc-a1d1-7eeb7ef6cb0e admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:01.047182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7ac155-857b-40bc-a1d1-7eeb7ef6cb0e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:01.056636 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 4/8] 10.222.0.39 () {58 vars in 1324 bytes} [Fri Sep 30 09:26:01 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 30 09:26:01.061928 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfacbf0-0ae8-4253-945b-1d5072a9e6c7 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:01.071135 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfacbf0-0ae8-4253-945b-1d5072a9e6c7 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:01.075486 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf97658b-9059-4fd2-9647-a9bdafdd0e33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:01.076108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf97658b-9059-4fd2-9647-a9bdafdd0e33 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:01.076392 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf97658b-9059-4fd2-9647-a9bdafdd0e33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:01.076667 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf97658b-9059-4fd2-9647-a9bdafdd0e33 admin admin] PATH_INFO: `/v3/users/767ee8fd8728440799a75a0b00b134c0` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:01.077963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf97658b-9059-4fd2-9647-a9bdafdd0e33 admin admin] RBAC: Authorizing `identity:get_user(user_id=767ee8fd8728440799a75a0b00b134c0)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:01.079895 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf97658b-9059-4fd2-9647-a9bdafdd0e33 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:01.081402 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 5/9] 10.222.0.39 () {56 vars in 1198 bytes} [Fri Sep 30 09:26:01 2022] GET /identity/v3/users/767ee8fd8728440799a75a0b00b134c0 => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:03.276055 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cce930-c6ce-4733-a5c9-6cc948062efa None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:03.276583 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cce930-c6ce-4733-a5c9-6cc948062efa None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:03.276583 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cce930-c6ce-4733-a5c9-6cc948062efa None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:03.277913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 5/10] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:03.283337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db20ec7b-9804-4ea6-a8f7-f5f3c2e8e9b7 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:03.283771 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db20ec7b-9804-4ea6-a8f7-f5f3c2e8e9b7 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:03.283875 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db20ec7b-9804-4ea6-a8f7-f5f3c2e8e9b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:03.297871 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc941a6-67de-4b5a-b830-b29d80664392 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:03.298167 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc941a6-67de-4b5a-b830-b29d80664392 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:03.298446 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc941a6-67de-4b5a-b830-b29d80664392 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:03.299819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 6/11] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:03.301816 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-db20ec7b-9804-4ea6-a8f7-f5f3c2e8e9b7 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:03.303064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-db20ec7b-9804-4ea6-a8f7-f5f3c2e8e9b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:03.304736 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f98f9b-1ea5-457a-91fd-82813b5de14f None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:03.305027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f98f9b-1ea5-457a-91fd-82813b5de14f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:03.305306 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f98f9b-1ea5-457a-91fd-82813b5de14f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:03.347304 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG passlib.handlers.bcrypt [None req-78f98f9b-1ea5-457a-91fd-82813b5de14f None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81684) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 30 09:26:03.347707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG passlib.handlers.bcrypt [None req-78f98f9b-1ea5-457a-91fd-82813b5de14f None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81684) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 30 09:26:03.383650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 6/12] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:26:03.397253 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-78f98f9b-1ea5-457a-91fd-82813b5de14f None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:03.398894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-78f98f9b-1ea5-457a-91fd-82813b5de14f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:03.496642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 7/13] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:26:03.646164 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db20ec7b-9804-4ea6-a8f7-f5f3c2e8e9b7 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:03.670859 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:03.749437 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db20ec7b-9804-4ea6-a8f7-f5f3c2e8e9b7 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:03.769249 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f98f9b-1ea5-457a-91fd-82813b5de14f None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:03.784691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666e6df9-7ebb-4da9-bf94-759a3845acab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:03.785333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666e6df9-7ebb-4da9-bf94-759a3845acab None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:03.785621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666e6df9-7ebb-4da9-bf94-759a3845acab None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:03.785902 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666e6df9-7ebb-4da9-bf94-759a3845acab None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:03.797046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666e6df9-7ebb-4da9-bf94-759a3845acab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:03.806490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-666e6df9-7ebb-4da9-bf94-759a3845acab None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:26:03.806490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:03.806490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:03.806490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:03.806490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:03.806490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:03.806490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:03.806490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:03.806490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:03.806490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:03.806490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:03.806490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:03.806490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:03.806490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:03.806490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:03.806490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:03.807419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:03.807419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:03.807419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:03.807419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:03.807419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:03.807419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:03.807419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:03.807419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:03.807419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:03.807419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:03.807419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:03.807419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:03.807419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:03.807419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:03.807419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:03.807419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:03.808215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:03.808215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:03.808215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:03.808215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:03.808215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:03.808215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:03.808215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:03.808215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:03.808215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:03.808215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:03.808215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:26:03.808215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:03.808215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 7/14] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:26:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:03.813405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666e6df9-7ebb-4da9-bf94-759a3845acab None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:03.868053 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f98f9b-1ea5-457a-91fd-82813b5de14f None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:03.870344 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:03.895106 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:03.905091 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d664d22-547f-4f00-9074-299d01fb1997 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:03.905954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d664d22-547f-4f00-9074-299d01fb1997 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:03.906237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d664d22-547f-4f00-9074-299d01fb1997 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:03.906512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d664d22-547f-4f00-9074-299d01fb1997 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:03.912792 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666e6df9-7ebb-4da9-bf94-759a3845acab None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:03.952200 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358103e1-01aa-4cb0-bfcc-eb84b3ac3806 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:03.952808 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358103e1-01aa-4cb0-bfcc-eb84b3ac3806 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:03.953082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358103e1-01aa-4cb0-bfcc-eb84b3ac3806 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:03.953354 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358103e1-01aa-4cb0-bfcc-eb84b3ac3806 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:03.954015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358103e1-01aa-4cb0-bfcc-eb84b3ac3806 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:03.954316 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358103e1-01aa-4cb0-bfcc-eb84b3ac3806 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:03.955755 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358103e1-01aa-4cb0-bfcc-eb84b3ac3806 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:03.967280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 8/15] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:03.973334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358103e1-01aa-4cb0-bfcc-eb84b3ac3806 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:03.974318 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d664d22-547f-4f00-9074-299d01fb1997 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:03.975882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d664d22-547f-4f00-9074-299d01fb1997 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:04.071721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358103e1-01aa-4cb0-bfcc-eb84b3ac3806 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:04.072364 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 8/16] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 30 09:26:04.077828 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d664d22-547f-4f00-9074-299d01fb1997 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:04.080499 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:04.105617 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3419a55a-793d-48b7-90ed-3c970ce2b671 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:04.106242 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3419a55a-793d-48b7-90ed-3c970ce2b671 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:04.106525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3419a55a-793d-48b7-90ed-3c970ce2b671 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:04.106824 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3419a55a-793d-48b7-90ed-3c970ce2b671 None admin] PATH_INFO: `/v3/users/767ee8fd8728440799a75a0b00b134c0` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:04.107954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3419a55a-793d-48b7-90ed-3c970ce2b671 None admin] RBAC: Authorizing `identity:get_user(user_id=767ee8fd8728440799a75a0b00b134c0)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:04.109408 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3419a55a-793d-48b7-90ed-3c970ce2b671 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:04.110984 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 9/17] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:03 2022] GET /identity/v3/users/767ee8fd8728440799a75a0b00b134c0 => generated 247 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:04.116881 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3419a55a-793d-48b7-90ed-3c970ce2b671 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:04.131120 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:04.180309 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d664d22-547f-4f00-9074-299d01fb1997 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:04.213924 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3419a55a-793d-48b7-90ed-3c970ce2b671 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:04.216728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d84e85-9022-471f-bed4-386f746d4783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:04.217608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d84e85-9022-471f-bed4-386f746d4783 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:04.217886 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d84e85-9022-471f-bed4-386f746d4783 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:04.218159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d84e85-9022-471f-bed4-386f746d4783 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:04.236515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d84e85-9022-471f-bed4-386f746d4783 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:04.238521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d84e85-9022-471f-bed4-386f746d4783 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:04.247806 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d0a7a8-afdc-4645-a6a7-580d3a929204 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:04.248434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d0a7a8-afdc-4645-a6a7-580d3a929204 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:04.248722 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d0a7a8-afdc-4645-a6a7-580d3a929204 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:04.248996 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d0a7a8-afdc-4645-a6a7-580d3a929204 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:04.250114 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d0a7a8-afdc-4645-a6a7-580d3a929204 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:04.251588 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d0a7a8-afdc-4645-a6a7-580d3a929204 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:04.253169 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 10/18] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:26:04 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:04.257202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-c3d84e85-9022-471f-bed4-386f746d4783 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:26:04.257202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:04.257202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:04.257202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:04.257202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:04.257202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:04.257202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:04.257202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:04.257202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:04.257202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:04.257202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:04.257202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:04.257202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:26:04.257202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:26:04.257202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:26:04.257202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:26:04.258369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:04.258369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:04.258369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:04.258369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:04.258369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:04.258369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:04.258369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:04.258369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:04.258369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:04.258369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:04.258369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:04.258369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:04.258369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:04.258369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:04.258369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:04.258369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:04.258369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:26:04.259234 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:26:04.259234 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:26:04.259234 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:26:04.259234 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 9/19] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:26:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:26:04.259937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d0a7a8-afdc-4645-a6a7-580d3a929204 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:04.263960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d84e85-9022-471f-bed4-386f746d4783 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:04.318564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:04.340027 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:04.359684 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d0a7a8-afdc-4645-a6a7-580d3a929204 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:04.362343 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d84e85-9022-471f-bed4-386f746d4783 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:04.394297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770b0820-53b6-4e38-8a0c-f02ed93b8a05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:04.394927 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770b0820-53b6-4e38-8a0c-f02ed93b8a05 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:04.395215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770b0820-53b6-4e38-8a0c-f02ed93b8a05 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:04.395474 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b16de2-364c-41bd-9ac7-58c117805358 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:04.395573 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770b0820-53b6-4e38-8a0c-f02ed93b8a05 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:04.396074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b16de2-364c-41bd-9ac7-58c117805358 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:04.396203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770b0820-53b6-4e38-8a0c-f02ed93b8a05 None admin] RBAC: Adding query filter params (role.id=e874c311084c4c9fb7691bce461d319b, scope.domain.id=default, user.id=767ee8fd8728440799a75a0b00b134c0) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:04.396359 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b16de2-364c-41bd-9ac7-58c117805358 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:04.396513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770b0820-53b6-4e38-8a0c-f02ed93b8a05 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:04.396634 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b16de2-364c-41bd-9ac7-58c117805358 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:04.397272 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b16de2-364c-41bd-9ac7-58c117805358 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:04.397579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b16de2-364c-41bd-9ac7-58c117805358 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:04.397975 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770b0820-53b6-4e38-8a0c-f02ed93b8a05 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:04.399073 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b16de2-364c-41bd-9ac7-58c117805358 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:04.404605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 11/20] 10.222.0.39 () {56 vars in 1322 bytes} [Fri Sep 30 09:26:04 2022] GET /identity/v3/role_assignments?user.id=767ee8fd8728440799a75a0b00b134c0&scope.domain.id=default&role.id=e874c311084c4c9fb7691bce461d319b => generated 233 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:04.417910 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 10/21] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:26:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:06.836921 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ae1311-054b-4b90-862d-4b67a70be9cc None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:06.837978 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ae1311-054b-4b90-862d-4b67a70be9cc None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:06.838479 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ae1311-054b-4b90-862d-4b67a70be9cc None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:06.840318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 12/22] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:06.845407 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b64ed7-09a2-4cd6-ab17-c67a09fe7f87 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:06.845847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b64ed7-09a2-4cd6-ab17-c67a09fe7f87 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:06.846003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b64ed7-09a2-4cd6-ab17-c67a09fe7f87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:06.871893 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-67b64ed7-09a2-4cd6-ab17-c67a09fe7f87 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:06.873454 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-67b64ed7-09a2-4cd6-ab17-c67a09fe7f87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:06.962090 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 11/23] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:07.002806 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3222322-96fb-43c1-a00e-df97215ef325 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:07.003152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3222322-96fb-43c1-a00e-df97215ef325 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:07.003642 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3222322-96fb-43c1-a00e-df97215ef325 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:07.005461 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 13/24] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:07.011277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7287e5-5bde-4ef2-8ad2-6c60a0ebd35b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:07.011566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7287e5-5bde-4ef2-8ad2-6c60a0ebd35b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:07.012023 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7287e5-5bde-4ef2-8ad2-6c60a0ebd35b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:07.037230 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-8a7287e5-5bde-4ef2-8ad2-6c60a0ebd35b None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:07.038719 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8a7287e5-5bde-4ef2-8ad2-6c60a0ebd35b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:07.084004 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:07.109900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 12/25] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:26:07.412004 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3222322-96fb-43c1-a00e-df97215ef325 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:07.485859 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a8bd09-e80b-430a-ba77-24a46f554f5c None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:07.486722 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a8bd09-e80b-430a-ba77-24a46f554f5c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:07.487239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a8bd09-e80b-430a-ba77-24a46f554f5c None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:07.488869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 13/26] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:26:07.494705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70c6ead-3a87-4dfa-8b05-5781d914445a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:07.495310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70c6ead-3a87-4dfa-8b05-5781d914445a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:07.495799 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70c6ead-3a87-4dfa-8b05-5781d914445a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:07.530939 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c70c6ead-3a87-4dfa-8b05-5781d914445a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:07.535808 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c70c6ead-3a87-4dfa-8b05-5781d914445a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:07.571135 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3222322-96fb-43c1-a00e-df97215ef325 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:07.614530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415a9aa2-3074-491d-88a7-4a828c11eb8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:07.615215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415a9aa2-3074-491d-88a7-4a828c11eb8d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:07.615498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415a9aa2-3074-491d-88a7-4a828c11eb8d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:07.615774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415a9aa2-3074-491d-88a7-4a828c11eb8d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:07.624752 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 14/27] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:26:07.625891 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70c6ead-3a87-4dfa-8b05-5781d914445a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:07.630163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415a9aa2-3074-491d-88a7-4a828c11eb8d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:07.636021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-415a9aa2-3074-491d-88a7-4a828c11eb8d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:26:07.636021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:07.636021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:07.636021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:07.636021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:07.636021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:07.636021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:07.636021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:07.636021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:07.636021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:07.636021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:07.636021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:07.636021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:07.636021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:07.636021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:07.636021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:07.636942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:07.636942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:07.636942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:07.636942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:07.636942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:07.636942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:07.636942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:07.636942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:07.636942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:07.636942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:07.636942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:07.636942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:07.636942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:07.636942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:07.636942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:07.636942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:07.637776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:07.637776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:07.637776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:07.637776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:07.637776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:07.637776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:07.637776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:07.637776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:07.637776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:07.637776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:07.637776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:26:07.637776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:07.637776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 14/28] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:26:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:07.639039 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4f7264-7c93-456b-96e6-c190fe73d318 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:07.639039 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4f7264-7c93-456b-96e6-c190fe73d318 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:07.639793 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4f7264-7c93-456b-96e6-c190fe73d318 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:07.646464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 15/29] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:07.647629 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4f7264-7c93-456b-96e6-c190fe73d318 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:07.654901 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:07.749710 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70c6ead-3a87-4dfa-8b05-5781d914445a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:07.758509 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:07.778898 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4f7264-7c93-456b-96e6-c190fe73d318 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:07.789985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d230f07b-8ecd-47f4-bfd2-3adb1ec3c178 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:07.790910 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d230f07b-8ecd-47f4-bfd2-3adb1ec3c178 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:07.791434 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d230f07b-8ecd-47f4-bfd2-3adb1ec3c178 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:07.791863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d230f07b-8ecd-47f4-bfd2-3adb1ec3c178 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:07.803421 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d230f07b-8ecd-47f4-bfd2-3adb1ec3c178 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:07.805295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d230f07b-8ecd-47f4-bfd2-3adb1ec3c178 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:07.816351 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423d67b7-d469-411e-a353-58cc24ab4f54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:07.817072 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423d67b7-d469-411e-a353-58cc24ab4f54 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:07.817456 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423d67b7-d469-411e-a353-58cc24ab4f54 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:07.817830 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423d67b7-d469-411e-a353-58cc24ab4f54 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:07.818602 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423d67b7-d469-411e-a353-58cc24ab4f54 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:07.819036 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423d67b7-d469-411e-a353-58cc24ab4f54 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:07.820510 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423d67b7-d469-411e-a353-58cc24ab4f54 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:07.823973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-d230f07b-8ecd-47f4-bfd2-3adb1ec3c178 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:26:07.823973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:07.823973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:07.823973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:07.823973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:07.823973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:07.823973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:07.823973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:07.823973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:07.823973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:07.823973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:07.823973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:07.823973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:26:07.823973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:26:07.823973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:26:07.823973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:26:07.825016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:07.825016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:07.825016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:07.825016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:07.825016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:07.825016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:07.825016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:07.825016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:07.825016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:07.825016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:07.825016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:07.825016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:07.825016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:07.825016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:07.825016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:07.825016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:07.825016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:26:07.825875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:26:07.825875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:26:07.825875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:26:07.827661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 15/30] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:26:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:07.828969 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7efe6d3-0f24-4a59-b513-f40e8c00ec85 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:07.829472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7efe6d3-0f24-4a59-b513-f40e8c00ec85 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:07.829931 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7efe6d3-0f24-4a59-b513-f40e8c00ec85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:07.835763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 16/31] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:07.837366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423d67b7-d469-411e-a353-58cc24ab4f54 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:07.851341 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b7efe6d3-0f24-4a59-b513-f40e8c00ec85 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:07.853238 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b7efe6d3-0f24-4a59-b513-f40e8c00ec85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:07.904998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:07.940616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 16/32] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:07.941747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7efe6d3-0f24-4a59-b513-f40e8c00ec85 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:07.957247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423d67b7-d469-411e-a353-58cc24ab4f54 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:08.023557 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d700bb1-d257-417c-b3a5-431421ac2c3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:08.024509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d700bb1-d257-417c-b3a5-431421ac2c3a None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:08.024982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d700bb1-d257-417c-b3a5-431421ac2c3a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:08.025477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d700bb1-d257-417c-b3a5-431421ac2c3a None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:08.026313 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d700bb1-d257-417c-b3a5-431421ac2c3a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:08.027518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d700bb1-d257-417c-b3a5-431421ac2c3a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:08.031364 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d700bb1-d257-417c-b3a5-431421ac2c3a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:08.062383 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 17/33] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:26:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:08.068683 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d700bb1-d257-417c-b3a5-431421ac2c3a None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:08.088268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:08.102064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7efe6d3-0f24-4a59-b513-f40e8c00ec85 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:08.141775 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a11421a-e281-46b0-9c31-b84a49e0c5ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:08.142942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a11421a-e281-46b0-9c31-b84a49e0c5ed None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:08.143421 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a11421a-e281-46b0-9c31-b84a49e0c5ed None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:08.143984 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a11421a-e281-46b0-9c31-b84a49e0c5ed None admin] PATH_INFO: `/v3/users/767ee8fd8728440799a75a0b00b134c0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:08.145655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a11421a-e281-46b0-9c31-b84a49e0c5ed None admin] RBAC: Authorizing `identity:get_user(user_id=767ee8fd8728440799a75a0b00b134c0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:08.147599 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a11421a-e281-46b0-9c31-b84a49e0c5ed None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:08.149372 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 17/34] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:07 2022] GET /identity/v3/users/767ee8fd8728440799a75a0b00b134c0 => generated 247 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:08.154931 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a11421a-e281-46b0-9c31-b84a49e0c5ed None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:08.261474 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d700bb1-d257-417c-b3a5-431421ac2c3a None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:08.266184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a11421a-e281-46b0-9c31-b84a49e0c5ed None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:08.288740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:08.297762 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd88091-6a89-4f7f-b80f-6f86af992671 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:08.298387 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd88091-6a89-4f7f-b80f-6f86af992671 None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:08.306630 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd88091-6a89-4f7f-b80f-6f86af992671 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:08.306630 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd88091-6a89-4f7f-b80f-6f86af992671 None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:08.306630 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd88091-6a89-4f7f-b80f-6f86af992671 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:08.306630 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd88091-6a89-4f7f-b80f-6f86af992671 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:08.326239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba99aad9-5a12-442e-a033-f4e83b2da76f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:08.327078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba99aad9-5a12-442e-a033-f4e83b2da76f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:08.327406 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba99aad9-5a12-442e-a033-f4e83b2da76f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:08.327733 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba99aad9-5a12-442e-a033-f4e83b2da76f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:08.328886 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba99aad9-5a12-442e-a033-f4e83b2da76f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:08.330513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba99aad9-5a12-442e-a033-f4e83b2da76f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:08.332288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 18/35] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:26:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:08.332978 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba99aad9-5a12-442e-a033-f4e83b2da76f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:08.362126 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:08.450904 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba99aad9-5a12-442e-a033-f4e83b2da76f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:08.487992 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e1c157-0b62-4f67-80d4-66299b290c99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:08.488951 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e1c157-0b62-4f67-80d4-66299b290c99 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:08.489426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e1c157-0b62-4f67-80d4-66299b290c99 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:08.489924 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e1c157-0b62-4f67-80d4-66299b290c99 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:08.490819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e1c157-0b62-4f67-80d4-66299b290c99 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:08.492418 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e1c157-0b62-4f67-80d4-66299b290c99 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:08.516722 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 18/36] 10.222.0.39 () {60 vars in 1141 bytes} [Fri Sep 30 09:26:08 2022] POST /identity/v3/projects => generated 301 bytes in 449 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:26:08.517596 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd88091-6a89-4f7f-b80f-6f86af992671 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:08.737205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 19/37] 10.222.0.39 () {60 vars in 1131 bytes} [Fri Sep 30 09:26:08 2022] POST /identity/v3/roles => generated 219 bytes in 404 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:26:08.738637 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e1c157-0b62-4f67-80d4-66299b290c99 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:08.815011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd88091-6a89-4f7f-b80f-6f86af992671 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:08.838219 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:08.857282 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e1c157-0b62-4f67-80d4-66299b290c99 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:08.860414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5570b0-3f70-4153-96a2-f9dddfddf019 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:08.861187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5570b0-3f70-4153-96a2-f9dddfddf019 None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:08.861607 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5570b0-3f70-4153-96a2-f9dddfddf019 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:08.861980 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5570b0-3f70-4153-96a2-f9dddfddf019 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:08.862876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5570b0-3f70-4153-96a2-f9dddfddf019 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:08.864679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5570b0-3f70-4153-96a2-f9dddfddf019 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:08.897972 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a723a9a8-8f21-4614-82d6-4d089e82aff7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:08.898696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a723a9a8-8f21-4614-82d6-4d089e82aff7 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:08.898989 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a723a9a8-8f21-4614-82d6-4d089e82aff7 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:08.899260 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a723a9a8-8f21-4614-82d6-4d089e82aff7 None admin] PATH_INFO: `/v3/domains/default/users/767ee8fd8728440799a75a0b00b134c0/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:08.901395 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a723a9a8-8f21-4614-82d6-4d089e82aff7 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=767ee8fd8728440799a75a0b00b134c0, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:08.903037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a723a9a8-8f21-4614-82d6-4d089e82aff7 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:08.927895 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 19/38] 10.222.0.39 () {60 vars in 1131 bytes} [Fri Sep 30 09:26:08 2022] POST /identity/v3/roles => generated 225 bytes in 411 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:26:08.929245 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31d28a4-18dd-4596-bc77-659ab76f7600 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:08.929539 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31d28a4-18dd-4596-bc77-659ab76f7600 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:08.929815 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31d28a4-18dd-4596-bc77-659ab76f7600 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:08.931391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 20/39] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:08.932334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476bf501-3ab0-457d-a002-3d6e40a02546 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:08.932651 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476bf501-3ab0-457d-a002-3d6e40a02546 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:08.932886 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476bf501-3ab0-457d-a002-3d6e40a02546 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:08.934387 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 21/40] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:08 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:08.941688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40e340e-c6b5-425e-8a55-b7ca5d0b8e8c None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:08.942046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40e340e-c6b5-425e-8a55-b7ca5d0b8e8c None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:08.942252 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40e340e-c6b5-425e-8a55-b7ca5d0b8e8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:08.964064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-d40e340e-c6b5-425e-8a55-b7ca5d0b8e8c None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:08.965955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d40e340e-c6b5-425e-8a55-b7ca5d0b8e8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:08.995012 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 20/41] 10.222.0.39 () {58 vars in 1361 bytes} [Fri Sep 30 09:26:08 2022] PUT /identity/v3/domains/default/users/767ee8fd8728440799a75a0b00b134c0/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:26:08.996748 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5059e095-6721-442a-b122-1f3bb5850a34 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:08.997239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5059e095-6721-442a-b122-1f3bb5850a34 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:08.997761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5059e095-6721-442a-b122-1f3bb5850a34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:09.007424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:09.025413 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-5059e095-6721-442a-b122-1f3bb5850a34 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:09.027811 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5059e095-6721-442a-b122-1f3bb5850a34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:09.066881 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 22/42] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:09.149149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 21/43] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:09.356224 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40e340e-c6b5-425e-8a55-b7ca5d0b8e8c None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:09.429380 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5059e095-6721-442a-b122-1f3bb5850a34 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:09.456613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40e340e-c6b5-425e-8a55-b7ca5d0b8e8c None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:09.491079 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e312ff-d262-465c-a600-57df86fddfe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:09.491712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e312ff-d262-465c-a600-57df86fddfe0 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:09.491994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e312ff-d262-465c-a600-57df86fddfe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:09.492269 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e312ff-d262-465c-a600-57df86fddfe0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:09.493364 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e312ff-d262-465c-a600-57df86fddfe0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:09.494868 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e312ff-d262-465c-a600-57df86fddfe0 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:09.496431 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 23/44] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:26:09 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:09.501747 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e312ff-d262-465c-a600-57df86fddfe0 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:09.507907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:09.534399 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5059e095-6721-442a-b122-1f3bb5850a34 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:09.545000 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:09.571470 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff91a6b3-9493-48d4-84f1-50f01f74959f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:09.572083 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff91a6b3-9493-48d4-84f1-50f01f74959f None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:09.572365 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff91a6b3-9493-48d4-84f1-50f01f74959f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:09.572724 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff91a6b3-9493-48d4-84f1-50f01f74959f None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:09.573307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff91a6b3-9493-48d4-84f1-50f01f74959f None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:09.574738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff91a6b3-9493-48d4-84f1-50f01f74959f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:09.612318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e312ff-d262-465c-a600-57df86fddfe0 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:09.646542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cf888d-fbc7-4927-a138-0ed4d3f999bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:09.647174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cf888d-fbc7-4927-a138-0ed4d3f999bc None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:09.647459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cf888d-fbc7-4927-a138-0ed4d3f999bc None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:09.647736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cf888d-fbc7-4927-a138-0ed4d3f999bc None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:09.648408 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92cf888d-fbc7-4927-a138-0ed4d3f999bc None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:09.649931 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92cf888d-fbc7-4927-a138-0ed4d3f999bc None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:09.669649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 22/45] 10.222.0.39 () {60 vars in 1131 bytes} [Fri Sep 30 09:26:09 2022] POST /identity/v3/roles => generated 223 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:26:09.746007 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 24/46] 10.222.0.39 () {60 vars in 1141 bytes} [Fri Sep 30 09:26:09 2022] POST /identity/v3/projects => generated 312 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:26:11.221846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abe7b13-f295-4653-bdfe-01dbf4c6aa52 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:11.222245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abe7b13-f295-4653-bdfe-01dbf4c6aa52 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:11.222410 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abe7b13-f295-4653-bdfe-01dbf4c6aa52 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:11.223779 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 23/47] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:11.228958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e270d9-b486-426a-a922-2e412ad746c5 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:11.229390 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e270d9-b486-426a-a922-2e412ad746c5 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:11.229515 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e270d9-b486-426a-a922-2e412ad746c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:11.248509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-09e270d9-b486-426a-a922-2e412ad746c5 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:11.249692 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-09e270d9-b486-426a-a922-2e412ad746c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:11.315087 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 25/48] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:26:11.578966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abe7b13-f295-4653-bdfe-01dbf4c6aa52 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:11.783144 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abe7b13-f295-4653-bdfe-01dbf4c6aa52 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:11.819824 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cbfc8c-9318-4474-9bc8-e65c9ef04d5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:11.820468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cbfc8c-9318-4474-9bc8-e65c9ef04d5a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:11.820749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cbfc8c-9318-4474-9bc8-e65c9ef04d5a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:11.821026 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cbfc8c-9318-4474-9bc8-e65c9ef04d5a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:11.823015 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:11.842699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cbfc8c-9318-4474-9bc8-e65c9ef04d5a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:11.842699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-c8cbfc8c-9318-4474-9bc8-e65c9ef04d5a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:26:11.842699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:11.842699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:11.842699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:11.842699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:11.842699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:11.842699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:11.842699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:11.842699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:11.842699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:11.842699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:11.842699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:11.842699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:11.843487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:11.843487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:11.843487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:11.843487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:11.843487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:11.843487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:11.843487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:11.843487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:11.843487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:11.843487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:11.843487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:11.843487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:11.843487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:11.843487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:11.843487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:11.843487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:11.843487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:11.844317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:11.844317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:11.844317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:11.844317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:11.844317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:11.844317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:11.844317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:11.844317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:11.844317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:11.844317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:11.844317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:11.844317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:11.844317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:26:11.844317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:26:11.844317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 24/49] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:26:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:11.846946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e270d9-b486-426a-a922-2e412ad746c5 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:11.864919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:11.903579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97971dee-c4ae-428c-8a62-78058e161a11 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:11.903865 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97971dee-c4ae-428c-8a62-78058e161a11 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:11.904145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97971dee-c4ae-428c-8a62-78058e161a11 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:11.905634 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 25/50] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:11.911068 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3150df-5b6a-4075-b3c2-d54902d75139 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:11.911357 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3150df-5b6a-4075-b3c2-d54902d75139 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:11.911637 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3150df-5b6a-4075-b3c2-d54902d75139 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:11.929297 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ed3150df-5b6a-4075-b3c2-d54902d75139 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:11.930510 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ed3150df-5b6a-4075-b3c2-d54902d75139 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:11.945762 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e270d9-b486-426a-a922-2e412ad746c5 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:11.981022 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4384fae4-260c-45fb-a8b6-7ff92e00e18a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:11.981629 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4384fae4-260c-45fb-a8b6-7ff92e00e18a None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:11.981915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4384fae4-260c-45fb-a8b6-7ff92e00e18a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:11.982195 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4384fae4-260c-45fb-a8b6-7ff92e00e18a None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:11.982894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4384fae4-260c-45fb-a8b6-7ff92e00e18a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:11.983209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4384fae4-260c-45fb-a8b6-7ff92e00e18a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:11.984614 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4384fae4-260c-45fb-a8b6-7ff92e00e18a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:11.992560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 26/51] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:11.997590 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 26/52] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:26:11.998239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4384fae4-260c-45fb-a8b6-7ff92e00e18a None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:12.076565 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:12.095325 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4384fae4-260c-45fb-a8b6-7ff92e00e18a None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:12.127831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcfb88c-4943-4911-9dc4-48a83991d147 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:12.128461 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcfb88c-4943-4911-9dc4-48a83991d147 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:12.128746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcfb88c-4943-4911-9dc4-48a83991d147 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:12.129023 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcfb88c-4943-4911-9dc4-48a83991d147 None admin] PATH_INFO: `/v3/users/767ee8fd8728440799a75a0b00b134c0` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:12.130363 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afcfb88c-4943-4911-9dc4-48a83991d147 None admin] RBAC: Authorizing `identity:get_user(user_id=767ee8fd8728440799a75a0b00b134c0)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:12.132320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afcfb88c-4943-4911-9dc4-48a83991d147 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:12.134830 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 27/53] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:11 2022] GET /identity/v3/users/767ee8fd8728440799a75a0b00b134c0 => generated 247 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:12.140186 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3150df-5b6a-4075-b3c2-d54902d75139 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:12.238339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3150df-5b6a-4075-b3c2-d54902d75139 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:12.261419 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:12.268485 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcfb88c-4943-4911-9dc4-48a83991d147 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:12.272304 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11c9556-c99a-4f34-a705-6761971987ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:12.272950 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11c9556-c99a-4f34-a705-6761971987ef None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:12.273231 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11c9556-c99a-4f34-a705-6761971987ef None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:12.273554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11c9556-c99a-4f34-a705-6761971987ef None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:12.274675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11c9556-c99a-4f34-a705-6761971987ef None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:12.276136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11c9556-c99a-4f34-a705-6761971987ef None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:12.277693 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 27/54] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:26:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:12.283984 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11c9556-c99a-4f34-a705-6761971987ef None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:12.367191 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcfb88c-4943-4911-9dc4-48a83991d147 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:12.383449 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11c9556-c99a-4f34-a705-6761971987ef None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:12.401671 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0727e8f-438f-48d5-9f93-26747fa38e8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:12.404228 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0727e8f-438f-48d5-9f93-26747fa38e8c None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:12.404228 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0727e8f-438f-48d5-9f93-26747fa38e8c None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:12.404228 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0727e8f-438f-48d5-9f93-26747fa38e8c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:12.404228 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0727e8f-438f-48d5-9f93-26747fa38e8c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:12.405427 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0727e8f-438f-48d5-9f93-26747fa38e8c None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:12.407048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 28/55] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:26:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:12.412535 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0727e8f-438f-48d5-9f93-26747fa38e8c None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:12.416655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12eeae6-a3e3-4864-9734-4422f1d7dc18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:12.417259 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12eeae6-a3e3-4864-9734-4422f1d7dc18 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:12.417538 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12eeae6-a3e3-4864-9734-4422f1d7dc18 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:12.417808 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12eeae6-a3e3-4864-9734-4422f1d7dc18 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:12.418521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12eeae6-a3e3-4864-9734-4422f1d7dc18 None admin] RBAC: Adding query filter params (role.id=e874c311084c4c9fb7691bce461d319b, scope.domain.id=default, user.id=767ee8fd8728440799a75a0b00b134c0) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:12.418855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12eeae6-a3e3-4864-9734-4422f1d7dc18 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:12.420287 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12eeae6-a3e3-4864-9734-4422f1d7dc18 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:12.428034 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 28/56] 10.222.0.39 () {56 vars in 1322 bytes} [Fri Sep 30 09:26:12 2022] GET /identity/v3/role_assignments?user.id=767ee8fd8728440799a75a0b00b134c0&scope.domain.id=default&role.id=e874c311084c4c9fb7691bce461d319b => generated 530 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:12.434506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:12.519302 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0727e8f-438f-48d5-9f93-26747fa38e8c None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:12.553831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06af7d92-c3cf-4420-8856-6d1c7f7dfffe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:12.554549 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06af7d92-c3cf-4420-8856-6d1c7f7dfffe None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:12.554859 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06af7d92-c3cf-4420-8856-6d1c7f7dfffe None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:12.555132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06af7d92-c3cf-4420-8856-6d1c7f7dfffe None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:12.555795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06af7d92-c3cf-4420-8856-6d1c7f7dfffe None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:12.557280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06af7d92-c3cf-4420-8856-6d1c7f7dfffe None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:12.642384 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 29/57] 10.222.0.39 () {60 vars in 1140 bytes} [Fri Sep 30 09:26:12 2022] POST /identity/v3/projects => generated 298 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:26:14.808555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566b9134-3292-43ab-9414-96802be8bd1d None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:14.809065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566b9134-3292-43ab-9414-96802be8bd1d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:14.809147 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566b9134-3292-43ab-9414-96802be8bd1d None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:14.810434 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 29/58] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:14.815735 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193c99d1-0f2f-43ca-a747-0b1928f45d6e None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:14.816146 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193c99d1-0f2f-43ca-a747-0b1928f45d6e None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:14.816306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193c99d1-0f2f-43ca-a747-0b1928f45d6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:14.834664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-193c99d1-0f2f-43ca-a747-0b1928f45d6e None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:14.835875 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-193c99d1-0f2f-43ca-a747-0b1928f45d6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:14.896504 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:14.901600 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 30/59] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:15.170088 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566b9134-3292-43ab-9414-96802be8bd1d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:15.227683 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:15.271023 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566b9134-3292-43ab-9414-96802be8bd1d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:15.302645 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b84af7-1cfc-43f2-880e-740c5c020fd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:15.303267 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b84af7-1cfc-43f2-880e-740c5c020fd9 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:15.303549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b84af7-1cfc-43f2-880e-740c5c020fd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:15.303822 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b84af7-1cfc-43f2-880e-740c5c020fd9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:15.304916 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b84af7-1cfc-43f2-880e-740c5c020fd9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:15.306365 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b84af7-1cfc-43f2-880e-740c5c020fd9 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:15.307933 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 30/60] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:26:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:15.313446 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193c99d1-0f2f-43ca-a747-0b1928f45d6e None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:15.408581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193c99d1-0f2f-43ca-a747-0b1928f45d6e None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:15.440714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e35976-27f6-49be-8399-634ca89dd655 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:15.441259 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e35976-27f6-49be-8399-634ca89dd655 None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:15.441538 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e35976-27f6-49be-8399-634ca89dd655 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:15.441809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e35976-27f6-49be-8399-634ca89dd655 None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:15.442455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e35976-27f6-49be-8399-634ca89dd655 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:15.443930 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e35976-27f6-49be-8399-634ca89dd655 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:15.514297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 31/61] 10.222.0.39 () {60 vars in 1132 bytes} [Fri Sep 30 09:26:15 2022] POST /identity/v3/users => generated 275 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:26:18.707631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0addbefc-61b5-4a5e-81a5-5bb129c34b92 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:18.708136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0addbefc-61b5-4a5e-81a5-5bb129c34b92 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:18.708239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0addbefc-61b5-4a5e-81a5-5bb129c34b92 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:18.709524 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 31/62] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:18.714842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6146ef-114c-44e7-bc31-246dcfc43bac None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:18.715656 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6146ef-114c-44e7-bc31-246dcfc43bac None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:18.716137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6146ef-114c-44e7-bc31-246dcfc43bac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:18.743185 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-0d6146ef-114c-44e7-bc31-246dcfc43bac None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:18.744925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0d6146ef-114c-44e7-bc31-246dcfc43bac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:18.810359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:18.831901 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 32/63] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:18.880995 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbd79c1-9881-4e96-909b-4cca6ca8054a None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:18.881645 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbd79c1-9881-4e96-909b-4cca6ca8054a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:18.882133 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbd79c1-9881-4e96-909b-4cca6ca8054a None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:18.884045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 32/64] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:18.889424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6513de-58a6-4320-9b14-2dcf2b9463db None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:18.890214 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6513de-58a6-4320-9b14-2dcf2b9463db None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:18.890936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6513de-58a6-4320-9b14-2dcf2b9463db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:18.919943 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-5b6513de-58a6-4320-9b14-2dcf2b9463db None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:18.921140 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-5b6513de-58a6-4320-9b14-2dcf2b9463db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:19.012441 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 33/65] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:19.138362 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a738e7-4f9c-45ed-a94a-cbf5daac974f None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:19.139014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a738e7-4f9c-45ed-a94a-cbf5daac974f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:19.139511 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a738e7-4f9c-45ed-a94a-cbf5daac974f None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:19.141289 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 33/66] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:19.147279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d82820-d5e3-4d9b-baee-59d74de31235 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:19.147798 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d82820-d5e3-4d9b-baee-59d74de31235 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:19.148278 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d82820-d5e3-4d9b-baee-59d74de31235 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:19.166557 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-c1d82820-d5e3-4d9b-baee-59d74de31235 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:19.168085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c1d82820-d5e3-4d9b-baee-59d74de31235 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:19.205989 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a738e7-4f9c-45ed-a94a-cbf5daac974f None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:19.322060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a738e7-4f9c-45ed-a94a-cbf5daac974f None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:19.328784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 34/67] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:19.329825 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d82820-d5e3-4d9b-baee-59d74de31235 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:19.359001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:19.375286 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8b0632-c740-465f-b2f3-091f8810446d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:19.378295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8b0632-c740-465f-b2f3-091f8810446d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:19.378775 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8b0632-c740-465f-b2f3-091f8810446d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:19.379189 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8b0632-c740-465f-b2f3-091f8810446d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:19.399260 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8b0632-c740-465f-b2f3-091f8810446d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:19.404493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-6b8b0632-c740-465f-b2f3-091f8810446d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:19.404493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:19.404493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:19.404493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:19.404493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:19.404493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:19.404493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:19.404493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:19.404493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:19.404493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:19.404493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:19.404493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:19.404493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:19.404493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:19.404493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:19.404493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:19.405533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:19.405533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:19.405533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:19.405533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:19.405533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:19.405533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:19.405533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:19.405533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:19.405533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:19.405533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:19.405533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:19.405533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:19.405533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:19.405533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:19.405533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:19.405533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:19.406487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:19.406487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:19.406487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:19.406487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:19.406487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:19.406487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:19.406487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:19.406487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:19.406487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:19.406487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:19.406487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:19.406487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:26:19.422669 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 34/68] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:19 2022] GET /identity/v3/roles/member => generated 84 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:19.426639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8b0632-c740-465f-b2f3-091f8810446d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:19.448720 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:19.511639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d82820-d5e3-4d9b-baee-59d74de31235 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:19.547061 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60355245-0bcd-4828-a9f9-b8443de4546b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:19.547714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60355245-0bcd-4828-a9f9-b8443de4546b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:19.548004 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60355245-0bcd-4828-a9f9-b8443de4546b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:19.548281 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60355245-0bcd-4828-a9f9-b8443de4546b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:19.564048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60355245-0bcd-4828-a9f9-b8443de4546b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:19.569509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-60355245-0bcd-4828-a9f9-b8443de4546b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:26:19.569509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:19.569509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:19.569509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:19.569509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:19.569509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:19.569509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:19.569509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:19.569509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:19.569509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:19.569509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:19.569509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:19.569509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:19.569509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:19.569509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:19.569509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:19.570531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:19.570531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:19.570531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:19.570531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:19.570531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:19.570531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:19.570531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:19.570531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:19.570531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:19.570531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:19.570531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:19.570531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:19.570531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:19.570531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:19.570531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:19.570531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:19.571366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:19.571366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:19.571366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:19.571366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:19.571366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:19.571366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:19.571366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:19.571366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:19.571366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:19.571366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:19.571366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:26:19.571366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:19.571366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 35/69] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:26:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:19.573532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e30f1e-9a7d-46f7-a739-5b9d0e41a32e None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:19.573778 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e30f1e-9a7d-46f7-a739-5b9d0e41a32e None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:19.574061 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e30f1e-9a7d-46f7-a739-5b9d0e41a32e None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:19.575346 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 36/70] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:19.579888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c494655-7c67-4263-bf51-6c738c23538a None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:19.580362 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c494655-7c67-4263-bf51-6c738c23538a None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:19.580644 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c494655-7c67-4263-bf51-6c738c23538a None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:19.582331 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 37/71] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:19 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:19.583657 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8b0632-c740-465f-b2f3-091f8810446d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:19.587026 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c494655-7c67-4263-bf51-6c738c23538a None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:19.620974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794178aa-05aa-430f-a624-d2e9eb9082e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:19.621591 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794178aa-05aa-430f-a624-d2e9eb9082e2 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:19.621896 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794178aa-05aa-430f-a624-d2e9eb9082e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:19.622150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794178aa-05aa-430f-a624-d2e9eb9082e2 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:19.622832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794178aa-05aa-430f-a624-d2e9eb9082e2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:19.623138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794178aa-05aa-430f-a624-d2e9eb9082e2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:19.624549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794178aa-05aa-430f-a624-d2e9eb9082e2 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:19.637281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 35/72] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:19 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:19.638631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c729e60-b5db-489c-a618-6fc13d2b6d8b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:19.638891 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c729e60-b5db-489c-a618-6fc13d2b6d8b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:19.639170 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c729e60-b5db-489c-a618-6fc13d2b6d8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:19.656291 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7c729e60-b5db-489c-a618-6fc13d2b6d8b None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:19.657508 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7c729e60-b5db-489c-a618-6fc13d2b6d8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:19.693088 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c494655-7c67-4263-bf51-6c738c23538a None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:19.695390 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:19.700089 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:19.726621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 36/73] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:19.726621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef844db-02db-4c99-a0e2-0f27a6e304e3 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:19.726621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef844db-02db-4c99-a0e2-0f27a6e304e3 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:19.726621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef844db-02db-4c99-a0e2-0f27a6e304e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:19.734401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2032fc5f-c495-4891-baa5-7a06b4979b3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:19.735037 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2032fc5f-c495-4891-baa5-7a06b4979b3c None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:19.735316 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2032fc5f-c495-4891-baa5-7a06b4979b3c None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:19.735587 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2032fc5f-c495-4891-baa5-7a06b4979b3c None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:19.736995 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2032fc5f-c495-4891-baa5-7a06b4979b3c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:19.737164 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2032fc5f-c495-4891-baa5-7a06b4979b3c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:19.738535 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2032fc5f-c495-4891-baa5-7a06b4979b3c None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:19.751648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 38/74] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:19.751648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2032fc5f-c495-4891-baa5-7a06b4979b3c None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:19.752436 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-1ef844db-02db-4c99-a0e2-0f27a6e304e3 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:19.754567 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1ef844db-02db-4c99-a0e2-0f27a6e304e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:19.827629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 37/75] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:19.828695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef844db-02db-4c99-a0e2-0f27a6e304e3 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:19.865712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2032fc5f-c495-4891-baa5-7a06b4979b3c None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:19.900254 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c4f61e-5e67-4bc7-981f-3f6ca16f33a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:19.900893 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c4f61e-5e67-4bc7-981f-3f6ca16f33a8 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:19.901183 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c4f61e-5e67-4bc7-981f-3f6ca16f33a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:19.901467 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c4f61e-5e67-4bc7-981f-3f6ca16f33a8 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:19.912436 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c4f61e-5e67-4bc7-981f-3f6ca16f33a8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:19.918525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-17c4f61e-5e67-4bc7-981f-3f6ca16f33a8 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:26:19.918525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:19.918525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:19.918525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:19.918525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:19.918525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:19.918525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:19.918525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:19.918525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:19.918525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:19.918525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:19.918525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:19.918525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:19.918525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:19.918525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:19.918525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:19.919433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:19.919433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:19.919433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:19.919433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:19.919433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:19.919433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:19.919433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:19.919433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:19.919433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:19.919433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:19.919433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:19.919433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:19.919433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:19.919433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:19.919433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:19.919433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:19.920215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:19.920215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:19.920215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:19.920215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:19.920215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:19.920215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:19.920215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:19.920215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:19.920215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:19.920215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:19.920215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:26:19.920215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:19.921102 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 39/76] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:26:19 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:19.921613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c4f61e-5e67-4bc7-981f-3f6ca16f33a8 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:19.951130 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:19.962099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef844db-02db-4c99-a0e2-0f27a6e304e3 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:19.972110 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:20.004058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f01171-8da0-49f7-a201-36b8c805f663 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:20.007737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f01171-8da0-49f7-a201-36b8c805f663 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:20.007737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f01171-8da0-49f7-a201-36b8c805f663 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:20.007737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f01171-8da0-49f7-a201-36b8c805f663 None admin] PATH_INFO: `/v3/users/dd33c1df1ffc427888650d2537b8f6d7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:20.023994 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f01171-8da0-49f7-a201-36b8c805f663 None admin] RBAC: Authorizing `identity:get_user(user_id=dd33c1df1ffc427888650d2537b8f6d7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:20.025452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f01171-8da0-49f7-a201-36b8c805f663 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:20.025780 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c4f61e-5e67-4bc7-981f-3f6ca16f33a8 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:20.027284 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 38/77] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:19 2022] GET /identity/v3/users/dd33c1df1ffc427888650d2537b8f6d7 => generated 275 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:20.027957 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f01171-8da0-49f7-a201-36b8c805f663 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:20.067827 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44b851d-d731-42fb-a799-15c133d8a737 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:20.068461 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44b851d-d731-42fb-a799-15c133d8a737 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:20.068741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44b851d-d731-42fb-a799-15c133d8a737 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:20.069014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44b851d-d731-42fb-a799-15c133d8a737 None admin] PATH_INFO: `/v3/users/767ee8fd8728440799a75a0b00b134c0` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:20.070264 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44b851d-d731-42fb-a799-15c133d8a737 None admin] RBAC: Authorizing `identity:get_user(user_id=767ee8fd8728440799a75a0b00b134c0)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:20.071751 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44b851d-d731-42fb-a799-15c133d8a737 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:20.074679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 40/78] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:19 2022] GET /identity/v3/users/767ee8fd8728440799a75a0b00b134c0 => generated 247 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:20.074679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44b851d-d731-42fb-a799-15c133d8a737 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:20.135017 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f01171-8da0-49f7-a201-36b8c805f663 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:20.168184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72e0cf9-11a1-493c-b6c2-89704b7f18d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:20.168476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:20.169153 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72e0cf9-11a1-493c-b6c2-89704b7f18d4 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:20.169602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72e0cf9-11a1-493c-b6c2-89704b7f18d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:20.169886 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72e0cf9-11a1-493c-b6c2-89704b7f18d4 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:20.170539 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72e0cf9-11a1-493c-b6c2-89704b7f18d4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:20.170878 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72e0cf9-11a1-493c-b6c2-89704b7f18d4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:20.172256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72e0cf9-11a1-493c-b6c2-89704b7f18d4 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:20.175291 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44b851d-d731-42fb-a799-15c133d8a737 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:20.182647 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 39/79] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:26:20 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:20.183462 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72e0cf9-11a1-493c-b6c2-89704b7f18d4 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:20.210146 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b0d6c0-c464-403f-8648-e17df908c07a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:20.210824 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b0d6c0-c464-403f-8648-e17df908c07a None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:20.211112 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b0d6c0-c464-403f-8648-e17df908c07a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:20.211232 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:20.211448 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b0d6c0-c464-403f-8648-e17df908c07a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:20.213053 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b0d6c0-c464-403f-8648-e17df908c07a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:20.214536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b0d6c0-c464-403f-8648-e17df908c07a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:20.216128 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 41/80] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:26:20 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:20.216890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b0d6c0-c464-403f-8648-e17df908c07a None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:20.283379 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72e0cf9-11a1-493c-b6c2-89704b7f18d4 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:20.315223 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b0d6c0-c464-403f-8648-e17df908c07a None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:20.315865 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024b9c12-ffe8-4862-ba17-92d7378105aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:20.316478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024b9c12-ffe8-4862-ba17-92d7378105aa None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:20.316766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024b9c12-ffe8-4862-ba17-92d7378105aa None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:20.317035 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024b9c12-ffe8-4862-ba17-92d7378105aa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:20.349660 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e9acc7-bc3e-4d49-ba34-cc90b717da0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:20.350254 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e9acc7-bc3e-4d49-ba34-cc90b717da0b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:20.350536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e9acc7-bc3e-4d49-ba34-cc90b717da0b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:20.350844 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e9acc7-bc3e-4d49-ba34-cc90b717da0b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:20.379034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:20.383197 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024b9c12-ffe8-4862-ba17-92d7378105aa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:20.384669 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024b9c12-ffe8-4862-ba17-92d7378105aa None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:20.411113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:20.416479 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e9acc7-bc3e-4d49-ba34-cc90b717da0b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:20.417949 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e9acc7-bc3e-4d49-ba34-cc90b717da0b None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:20.481998 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 40/81] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:26:20.482855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024b9c12-ffe8-4862-ba17-92d7378105aa None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:20.518066 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 42/82] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:26:20.519151 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e9acc7-bc3e-4d49-ba34-cc90b717da0b None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:20.577120 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:20.579201 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024b9c12-ffe8-4862-ba17-92d7378105aa None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:20.603862 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:20.612649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a2330b-dbd1-4e2d-9ced-d8587a54d8dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:20.615765 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e9acc7-bc3e-4d49-ba34-cc90b717da0b None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:20.617660 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a2330b-dbd1-4e2d-9ced-d8587a54d8dd None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:20.617957 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a2330b-dbd1-4e2d-9ced-d8587a54d8dd None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:20.618238 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a2330b-dbd1-4e2d-9ced-d8587a54d8dd None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:20.629250 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a2330b-dbd1-4e2d-9ced-d8587a54d8dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:20.634797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-f8a2330b-dbd1-4e2d-9ced-d8587a54d8dd None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:20.634797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:20.634797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:20.634797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:20.634797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:20.634797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:20.634797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:20.634797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:20.634797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:20.634797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:20.634797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:20.634797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:20.634797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:20.634797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:20.634797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:20.634797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:20.635665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:20.635665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:20.635665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:20.635665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:20.635665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:20.635665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:20.635665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:20.635665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:20.635665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:20.635665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:20.635665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:20.635665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:20.635665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:20.635665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:20.635665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:20.635665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:20.636708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:20.636708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:20.636708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:20.636708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:20.636708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:20.636708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:20.636708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:20.636708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:20.636708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:20.636708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:20.636708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:20.636708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:26:20.636708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 41/83] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:20 2022] GET /identity/v3/roles/member => generated 84 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:20.636708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a2330b-dbd1-4e2d-9ced-d8587a54d8dd None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:20.652138 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0aa241-df8b-448e-8f84-f428f6f6a1a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:20.652772 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0aa241-df8b-448e-8f84-f428f6f6a1a6 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:20.653057 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0aa241-df8b-448e-8f84-f428f6f6a1a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:20.653334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0aa241-df8b-448e-8f84-f428f6f6a1a6 None admin] PATH_INFO: `/v3/users/dd33c1df1ffc427888650d2537b8f6d7` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:20.654444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0aa241-df8b-448e-8f84-f428f6f6a1a6 None admin] RBAC: Authorizing `identity:get_user(user_id=dd33c1df1ffc427888650d2537b8f6d7)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:20.655929 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0aa241-df8b-448e-8f84-f428f6f6a1a6 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:20.657488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 43/84] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:20 2022] GET /identity/v3/users/dd33c1df1ffc427888650d2537b8f6d7 => generated 275 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:20.658282 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0aa241-df8b-448e-8f84-f428f6f6a1a6 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:20.739592 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a2330b-dbd1-4e2d-9ced-d8587a54d8dd None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:20.755519 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0aa241-df8b-448e-8f84-f428f6f6a1a6 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:20.772661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0417dbf-5160-41f2-ad5c-7eafe8490ae3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:20.773266 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0417dbf-5160-41f2-ad5c-7eafe8490ae3 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:20.773545 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0417dbf-5160-41f2-ad5c-7eafe8490ae3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:20.773817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0417dbf-5160-41f2-ad5c-7eafe8490ae3 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:20.774454 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0417dbf-5160-41f2-ad5c-7eafe8490ae3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:20.775937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0417dbf-5160-41f2-ad5c-7eafe8490ae3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:20.782984 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:20.788462 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ab638e-3d4d-459b-bc32-e98fb6210af8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:20.789100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ab638e-3d4d-459b-bc32-e98fb6210af8 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:20.789383 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ab638e-3d4d-459b-bc32-e98fb6210af8 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:20.789662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ab638e-3d4d-459b-bc32-e98fb6210af8 None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:20.790803 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ab638e-3d4d-459b-bc32-e98fb6210af8 None admin] RBAC: Authorizing `identity:get_project(project_id=333c93c6a3ef48b6806ea0d5f14ce211)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:20.792258 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ab638e-3d4d-459b-bc32-e98fb6210af8 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:20.793865 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 44/85] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:20 2022] GET /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211 => generated 298 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:20.794638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ab638e-3d4d-459b-bc32-e98fb6210af8 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:20.874856 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 42/86] 10.222.0.39 () {60 vars in 1132 bytes} [Fri Sep 30 09:26:20 2022] POST /identity/v3/users => generated 289 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:26:20.875722 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0417dbf-5160-41f2-ad5c-7eafe8490ae3 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:20.902280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ab638e-3d4d-459b-bc32-e98fb6210af8 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:20.924854 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:20.947557 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c69688-b917-4989-b97f-4cf90b8e7cbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:20.948219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c69688-b917-4989-b97f-4cf90b8e7cbd None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:20.948505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c69688-b917-4989-b97f-4cf90b8e7cbd None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:20.948777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c69688-b917-4989-b97f-4cf90b8e7cbd None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:20.949992 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c69688-b917-4989-b97f-4cf90b8e7cbd None admin] RBAC: Authorizing `identity:get_project(project_id=333c93c6a3ef48b6806ea0d5f14ce211)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:20.951533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c69688-b917-4989-b97f-4cf90b8e7cbd None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:20.958581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 45/87] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:20 2022] GET /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211 => generated 298 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:20.958581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c69688-b917-4989-b97f-4cf90b8e7cbd None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:20.993980 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0417dbf-5160-41f2-ad5c-7eafe8490ae3 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:21.031477 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c17668d-8fc5-4e72-b33a-7ee3d76f594f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:21.032119 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c17668d-8fc5-4e72-b33a-7ee3d76f594f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:21.032405 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c17668d-8fc5-4e72-b33a-7ee3d76f594f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:21.032708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c17668d-8fc5-4e72-b33a-7ee3d76f594f None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:21.033469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c17668d-8fc5-4e72-b33a-7ee3d76f594f None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:21.033782 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c17668d-8fc5-4e72-b33a-7ee3d76f594f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:21.035309 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c17668d-8fc5-4e72-b33a-7ee3d76f594f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:21.040605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:21.045308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 43/88] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:20 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:21.045308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c17668d-8fc5-4e72-b33a-7ee3d76f594f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:21.112907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c69688-b917-4989-b97f-4cf90b8e7cbd None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:21.154542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ef089d-11cf-4531-89e5-279a1dbdbfbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:21.155627 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ef089d-11cf-4531-89e5-279a1dbdbfbf None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:21.156134 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ef089d-11cf-4531-89e5-279a1dbdbfbf None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:21.156425 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ef089d-11cf-4531-89e5-279a1dbdbfbf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:21.163796 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c17668d-8fc5-4e72-b33a-7ee3d76f594f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:21.166978 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:21.212800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff35ae5c-4a87-4f79-95d1-4b9f62dfef2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:21.213417 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff35ae5c-4a87-4f79-95d1-4b9f62dfef2a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:21.213696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff35ae5c-4a87-4f79-95d1-4b9f62dfef2a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:21.213969 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff35ae5c-4a87-4f79-95d1-4b9f62dfef2a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:21.214725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff35ae5c-4a87-4f79-95d1-4b9f62dfef2a None admin] RBAC: Adding query filter params (role.id=025f58735650469d9fa6faca3e9e7998, scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211, user.id=dd33c1df1ffc427888650d2537b8f6d7) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:21.215032 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff35ae5c-4a87-4f79-95d1-4b9f62dfef2a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:21.216491 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff35ae5c-4a87-4f79-95d1-4b9f62dfef2a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:21.223529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 44/89] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:26:21 2022] GET /identity/v3/role_assignments?user.id=dd33c1df1ffc427888650d2537b8f6d7&scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211&role.id=025f58735650469d9fa6faca3e9e7998 => generated 259 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:21.224385 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff35ae5c-4a87-4f79-95d1-4b9f62dfef2a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:21.254247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ef089d-11cf-4531-89e5-279a1dbdbfbf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:21.256093 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ef089d-11cf-4531-89e5-279a1dbdbfbf None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:21.331522 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:21.343676 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff35ae5c-4a87-4f79-95d1-4b9f62dfef2a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:21.385091 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 46/90] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 431 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:26:21.387851 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ef089d-11cf-4531-89e5-279a1dbdbfbf None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:21.415386 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52233296-7d95-4273-b195-486ea7d79ad2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:21.416255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52233296-7d95-4273-b195-486ea7d79ad2 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:21.416679 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52233296-7d95-4273-b195-486ea7d79ad2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:21.417117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52233296-7d95-4273-b195-486ea7d79ad2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:21.417992 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52233296-7d95-4273-b195-486ea7d79ad2 None admin] RBAC: Adding query filter params (role.id=e874c311084c4c9fb7691bce461d319b, scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211, user.id=767ee8fd8728440799a75a0b00b134c0) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:21.418453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52233296-7d95-4273-b195-486ea7d79ad2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:21.420160 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52233296-7d95-4273-b195-486ea7d79ad2 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:21.437809 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 45/91] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:26:21 2022] GET /identity/v3/role_assignments?user.id=767ee8fd8728440799a75a0b00b134c0&scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211&role.id=e874c311084c4c9fb7691bce461d319b => generated 259 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:21.439071 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52233296-7d95-4273-b195-486ea7d79ad2 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:21.517564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:21.577014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ef089d-11cf-4531-89e5-279a1dbdbfbf None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:21.631407 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52233296-7d95-4273-b195-486ea7d79ad2 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:21.650933 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549ff15d-be1a-48e4-9ba5-c40a2dbcb869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:21.652045 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549ff15d-be1a-48e4-9ba5-c40a2dbcb869 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:21.652532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549ff15d-be1a-48e4-9ba5-c40a2dbcb869 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:21.653139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549ff15d-be1a-48e4-9ba5-c40a2dbcb869 None admin] PATH_INFO: `/v3/users/dd33c1df1ffc427888650d2537b8f6d7` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:21.654786 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549ff15d-be1a-48e4-9ba5-c40a2dbcb869 None admin] RBAC: Authorizing `identity:get_user(user_id=dd33c1df1ffc427888650d2537b8f6d7)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:21.656553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549ff15d-be1a-48e4-9ba5-c40a2dbcb869 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:21.659347 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 47/92] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:21 2022] GET /identity/v3/users/dd33c1df1ffc427888650d2537b8f6d7 => generated 275 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:21.665762 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549ff15d-be1a-48e4-9ba5-c40a2dbcb869 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:21.670509 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea7c9e7-eb77-4380-b4ed-2821b25a07bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:21.671219 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea7c9e7-eb77-4380-b4ed-2821b25a07bd None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:21.671610 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea7c9e7-eb77-4380-b4ed-2821b25a07bd None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:21.671943 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea7c9e7-eb77-4380-b4ed-2821b25a07bd None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:21.673677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea7c9e7-eb77-4380-b4ed-2821b25a07bd None admin] RBAC: Authorizing `identity:get_project(project_id=333c93c6a3ef48b6806ea0d5f14ce211)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:21.674726 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea7c9e7-eb77-4380-b4ed-2821b25a07bd None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:21.676590 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 46/93] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:21 2022] GET /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211 => generated 298 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:21.688143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea7c9e7-eb77-4380-b4ed-2821b25a07bd None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:21.732064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:21.815801 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea7c9e7-eb77-4380-b4ed-2821b25a07bd None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:21.851239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4104c023-401e-4a69-af34-6d0f5d291c12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:21.852062 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4104c023-401e-4a69-af34-6d0f5d291c12 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:21.852449 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4104c023-401e-4a69-af34-6d0f5d291c12 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:21.852796 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4104c023-401e-4a69-af34-6d0f5d291c12 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:21.853583 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4104c023-401e-4a69-af34-6d0f5d291c12 None admin] RBAC: Adding query filter params (role.id=da6797f7fa24429095283b460be5c792, scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211, user.id=dd33c1df1ffc427888650d2537b8f6d7) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:21.853982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4104c023-401e-4a69-af34-6d0f5d291c12 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:21.855558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4104c023-401e-4a69-af34-6d0f5d291c12 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:21.858955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:21.863655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 47/94] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:26:21 2022] GET /identity/v3/role_assignments?user.id=dd33c1df1ffc427888650d2537b8f6d7&scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211&role.id=da6797f7fa24429095283b460be5c792 => generated 259 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:21.869642 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549ff15d-be1a-48e4-9ba5-c40a2dbcb869 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:21.910630 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847f2fdf-abde-4dc8-b944-6b15c616203b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:21.911380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847f2fdf-abde-4dc8-b944-6b15c616203b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:21.911765 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847f2fdf-abde-4dc8-b944-6b15c616203b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:21.912130 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847f2fdf-abde-4dc8-b944-6b15c616203b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:22.003205 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847f2fdf-abde-4dc8-b944-6b15c616203b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:22.004732 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847f2fdf-abde-4dc8-b944-6b15c616203b None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:22.057260 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:22.143858 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 48/95] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 478 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:26:22.149992 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4104c023-401e-4a69-af34-6d0f5d291c12 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:22.279175 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4104c023-401e-4a69-af34-6d0f5d291c12 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:22.318188 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33ef4fc-58ce-4a42-b564-1822cc8bcf4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:22.318861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33ef4fc-58ce-4a42-b564-1822cc8bcf4e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:22.319180 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33ef4fc-58ce-4a42-b564-1822cc8bcf4e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:22.319459 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33ef4fc-58ce-4a42-b564-1822cc8bcf4e None admin] PATH_INFO: `/v3/projects/a8737ba920904c149557cbac6c1301c6` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:22.321156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33ef4fc-58ce-4a42-b564-1822cc8bcf4e None admin] RBAC: Authorizing `identity:get_project(project_id=a8737ba920904c149557cbac6c1301c6)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:22.322630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33ef4fc-58ce-4a42-b564-1822cc8bcf4e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:22.325247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 48/96] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:22 2022] GET /identity/v3/projects/a8737ba920904c149557cbac6c1301c6 => generated 312 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:22.330802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847f2fdf-abde-4dc8-b944-6b15c616203b None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:22.456055 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847f2fdf-abde-4dc8-b944-6b15c616203b None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:22.499828 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc895b6-10c3-43c0-b8e8-0e00cbfad2fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:22.500442 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc895b6-10c3-43c0-b8e8-0e00cbfad2fa None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:22.500789 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc895b6-10c3-43c0-b8e8-0e00cbfad2fa None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:22.501120 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc895b6-10c3-43c0-b8e8-0e00cbfad2fa None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:22.502513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc895b6-10c3-43c0-b8e8-0e00cbfad2fa None admin] RBAC: Adding query filter params (role.id=025f58735650469d9fa6faca3e9e7998, scope.project.id=a8737ba920904c149557cbac6c1301c6, user.id=dd33c1df1ffc427888650d2537b8f6d7) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:22.502891 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc895b6-10c3-43c0-b8e8-0e00cbfad2fa None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:22.504362 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc895b6-10c3-43c0-b8e8-0e00cbfad2fa None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:22.506119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:22.514705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 49/97] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:26:22 2022] GET /identity/v3/role_assignments?user.id=dd33c1df1ffc427888650d2537b8f6d7&scope.project.id=a8737ba920904c149557cbac6c1301c6&role.id=025f58735650469d9fa6faca3e9e7998 => generated 259 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:23.866636 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cadff6-920a-48c2-bca1-7c5ed441e6b4 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:23.867662 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cadff6-920a-48c2-bca1-7c5ed441e6b4 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:23.868153 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cadff6-920a-48c2-bca1-7c5ed441e6b4 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:23.869707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 49/98] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:23.875175 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bdf8a3-fb7e-4205-904b-096331b54cbf None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:23.876059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bdf8a3-fb7e-4205-904b-096331b54cbf None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:23.877108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bdf8a3-fb7e-4205-904b-096331b54cbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:23.896314 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-88bdf8a3-fb7e-4205-904b-096331b54cbf None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:23.897846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-88bdf8a3-fb7e-4205-904b-096331b54cbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:23.983978 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 50/99] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:24.203879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965c805e-55db-45fe-8a1c-85c65789abdf None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:24.204612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965c805e-55db-45fe-8a1c-85c65789abdf None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:24.205099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965c805e-55db-45fe-8a1c-85c65789abdf None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:24.206730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 50/100] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:24.223504 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0ed454-d0e4-4e3b-ba55-306a16ae1614 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:24.224113 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0ed454-d0e4-4e3b-ba55-306a16ae1614 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:24.224593 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0ed454-d0e4-4e3b-ba55-306a16ae1614 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:24.250499 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-0c0ed454-d0e4-4e3b-ba55-306a16ae1614 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:24.252089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0c0ed454-d0e4-4e3b-ba55-306a16ae1614 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:24.347889 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 51/101] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:24.440032 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965c805e-55db-45fe-8a1c-85c65789abdf None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:24.479585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:24.536142 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde2b8d8-38e3-4580-8e73-1b5606a677c5 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:24.536777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde2b8d8-38e3-4580-8e73-1b5606a677c5 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:24.537263 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde2b8d8-38e3-4580-8e73-1b5606a677c5 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:24.538932 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 52/102] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:24.548550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ec5b23-60b6-4208-bb74-ee55fc7d1874 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:24.548838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ec5b23-60b6-4208-bb74-ee55fc7d1874 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:24.549116 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ec5b23-60b6-4208-bb74-ee55fc7d1874 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:24.567628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-d7ec5b23-60b6-4208-bb74-ee55fc7d1874 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:24.569061 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d7ec5b23-60b6-4208-bb74-ee55fc7d1874 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:24.595644 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965c805e-55db-45fe-8a1c-85c65789abdf None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:24.628644 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:24.639502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 53/103] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:26:24.658094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ff7470-2a02-47ce-aaee-f1636191347d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:24.658779 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ff7470-2a02-47ce-aaee-f1636191347d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:24.659115 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ff7470-2a02-47ce-aaee-f1636191347d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:24.659392 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ff7470-2a02-47ce-aaee-f1636191347d None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:24.670787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ff7470-2a02-47ce-aaee-f1636191347d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:24.676487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-94ff7470-2a02-47ce-aaee-f1636191347d None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 30 09:26:24.676487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:24.676487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:24.676487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:24.676487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:24.676487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:24.676487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:24.676487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:24.676487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:24.676487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:24.676487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:24.676487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:24.676487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:24.676487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:24.676487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:24.676487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:24.677351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:24.677351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:24.677351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:24.677351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:24.677351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:24.677351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:24.677351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:24.677351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:24.677351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:24.677351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:24.677351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:24.677351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:24.677351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:24.677351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:24.677351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:24.677351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:24.678145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:24.678145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:24.678145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:24.678145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:24.678145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:24.678145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:24.678145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:24.678145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:24.678145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:24.678145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:24.678145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 30 09:26:24.678145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:26:24.678145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 51/104] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:24 2022] GET /identity/v3/roles/reader => generated 84 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:24.688619 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ec5b23-60b6-4208-bb74-ee55fc7d1874 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:24.822117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ec5b23-60b6-4208-bb74-ee55fc7d1874 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:24.862559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40fd9fc2-da15-424c-b3aa-2c0d5cf9e275 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:24.863579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fd9fc2-da15-424c-b3aa-2c0d5cf9e275 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:24.864140 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fd9fc2-da15-424c-b3aa-2c0d5cf9e275 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:24.864607 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fd9fc2-da15-424c-b3aa-2c0d5cf9e275 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:24.865472 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40fd9fc2-da15-424c-b3aa-2c0d5cf9e275 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:24.865640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b639ac-797b-487c-a679-257df535d263 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:24.865888 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b639ac-797b-487c-a679-257df535d263 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:24.866169 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b639ac-797b-487c-a679-257df535d263 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:24.866536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40fd9fc2-da15-424c-b3aa-2c0d5cf9e275 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:24.867569 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 52/105] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:24.868725 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40fd9fc2-da15-424c-b3aa-2c0d5cf9e275 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:24.872469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b04f1e-0fdb-404f-b9a3-bcb20abfe54f None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:24.872761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b04f1e-0fdb-404f-b9a3-bcb20abfe54f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:24.873042 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b04f1e-0fdb-404f-b9a3-bcb20abfe54f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:24.884312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 54/106] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:24 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:24.890070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40fd9fc2-da15-424c-b3aa-2c0d5cf9e275 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:24.910859 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-01b04f1e-0fdb-404f-b9a3-bcb20abfe54f None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:24.913294 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-01b04f1e-0fdb-404f-b9a3-bcb20abfe54f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:24.982981 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:25.047052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 53/107] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:26:25.048080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b04f1e-0fdb-404f-b9a3-bcb20abfe54f None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:25.074766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:25.082622 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40fd9fc2-da15-424c-b3aa-2c0d5cf9e275 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:25.159841 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b04f1e-0fdb-404f-b9a3-bcb20abfe54f None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:25.162767 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d682a52-5ffe-4b2e-bbeb-3535d3eee7be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:25.163418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d682a52-5ffe-4b2e-bbeb-3535d3eee7be None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:25.163705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d682a52-5ffe-4b2e-bbeb-3535d3eee7be None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:25.163990 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d682a52-5ffe-4b2e-bbeb-3535d3eee7be None admin] PATH_INFO: `/v3/users/41f973abcdc243e1b159643b7953c768` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:25.195968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d682a52-5ffe-4b2e-bbeb-3535d3eee7be None admin] RBAC: Authorizing `identity:get_user(user_id=41f973abcdc243e1b159643b7953c768)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:25.197463 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d682a52-5ffe-4b2e-bbeb-3535d3eee7be None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:25.199188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 55/108] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:24 2022] GET /identity/v3/users/41f973abcdc243e1b159643b7953c768 => generated 289 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:25.200083 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d682a52-5ffe-4b2e-bbeb-3535d3eee7be None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:25.218891 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d6a2ed-9a3f-4cc4-b40f-748b4509f2e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:25.219847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d6a2ed-9a3f-4cc4-b40f-748b4509f2e1 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:25.220663 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d6a2ed-9a3f-4cc4-b40f-748b4509f2e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:25.220941 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d6a2ed-9a3f-4cc4-b40f-748b4509f2e1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:25.228363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d6a2ed-9a3f-4cc4-b40f-748b4509f2e1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:25.229840 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d6a2ed-9a3f-4cc4-b40f-748b4509f2e1 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:25.231722 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 54/109] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:26:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:25.233472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d6a2ed-9a3f-4cc4-b40f-748b4509f2e1 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:25.348062 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d682a52-5ffe-4b2e-bbeb-3535d3eee7be None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:25.368331 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:25.413593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:25.419093 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f1de1c-63f3-4517-94e3-14d90881c659 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:25.419898 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f1de1c-63f3-4517-94e3-14d90881c659 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:25.420335 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f1de1c-63f3-4517-94e3-14d90881c659 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:25.420766 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f1de1c-63f3-4517-94e3-14d90881c659 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:25.423529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d6a2ed-9a3f-4cc4-b40f-748b4509f2e1 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:25.431887 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f1de1c-63f3-4517-94e3-14d90881c659 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:25.436962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-02f1de1c-63f3-4517-94e3-14d90881c659 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:25.436962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:25.436962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:25.436962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:25.436962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:25.436962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:25.436962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:25.436962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:25.436962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:25.436962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:25.436962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:25.436962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:25.436962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:25.436962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:25.436962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:25.436962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:25.438092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:25.438092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:25.438092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:25.438092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:25.438092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:25.438092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:25.438092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:25.438092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:25.438092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:25.438092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:25.438092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:25.438092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:25.438092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:25.438092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:25.438092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:25.438092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:25.439145 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:25.439145 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:25.439145 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:25.439145 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:25.439145 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:25.439145 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:25.439145 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:25.439145 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:25.439145 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:25.439145 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:25.439145 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:25.439145 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:25.441019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 56/110] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:25 2022] GET /identity/v3/roles/member => generated 84 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:25.441907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f1de1c-63f3-4517-94e3-14d90881c659 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:25.459094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38646d37-771a-4649-87a9-d73c7e06e71a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:25.459713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38646d37-771a-4649-87a9-d73c7e06e71a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:25.459993 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38646d37-771a-4649-87a9-d73c7e06e71a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:25.460267 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38646d37-771a-4649-87a9-d73c7e06e71a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:25.535730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38646d37-771a-4649-87a9-d73c7e06e71a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:25.537256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38646d37-771a-4649-87a9-d73c7e06e71a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:25.549297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f1de1c-63f3-4517-94e3-14d90881c659 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:25.587062 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cd3dac-0e99-4f00-a27a-60f4950cba1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:25.587702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cd3dac-0e99-4f00-a27a-60f4950cba1e None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:25.587992 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cd3dac-0e99-4f00-a27a-60f4950cba1e None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:25.588298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cd3dac-0e99-4f00-a27a-60f4950cba1e None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:25.588956 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cd3dac-0e99-4f00-a27a-60f4950cba1e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:25.590475 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cd3dac-0e99-4f00-a27a-60f4950cba1e None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:25.631755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:25.641744 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 55/111] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:26:25.643525 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218f7bfb-2bc9-49aa-aa5e-9c5161bdec93 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:25.643814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218f7bfb-2bc9-49aa-aa5e-9c5161bdec93 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:25.644084 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218f7bfb-2bc9-49aa-aa5e-9c5161bdec93 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:25.645311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 56/112] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:25.646004 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218f7bfb-2bc9-49aa-aa5e-9c5161bdec93 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:25.705295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 57/113] 10.222.0.39 () {60 vars in 1141 bytes} [Fri Sep 30 09:26:25 2022] POST /identity/v3/projects => generated 302 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:26:25.705295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cd3dac-0e99-4f00-a27a-60f4950cba1e None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:25.720950 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:25.757190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218f7bfb-2bc9-49aa-aa5e-9c5161bdec93 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:25.797130 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8536740-1c7b-4d81-b865-fe72667e75e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:25.797797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8536740-1c7b-4d81-b865-fe72667e75e1 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:25.798114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8536740-1c7b-4d81-b865-fe72667e75e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:25.798385 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8536740-1c7b-4d81-b865-fe72667e75e1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:25.810413 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cd3dac-0e99-4f00-a27a-60f4950cba1e None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:25.810836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8536740-1c7b-4d81-b865-fe72667e75e1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:25.816935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-e8536740-1c7b-4d81-b865-fe72667e75e1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:26:25.816935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:25.816935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:25.816935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:25.816935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:25.816935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:25.816935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:25.816935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:25.816935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:25.816935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:25.816935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:25.816935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:25.816935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:25.816935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:25.816935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:25.816935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:25.818107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:25.818107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:25.818107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:25.818107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:25.818107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:25.818107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:25.818107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:25.818107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:25.818107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:25.818107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:25.818107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:25.818107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:25.818107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:25.818107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:25.818107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:25.818107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:25.820030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:25.820030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:25.820030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:25.820030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:25.820030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:25.820030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:25.820030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:25.820030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:25.820030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:25.820030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:25.820030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:26:25.820030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:26:25.820030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 57/114] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:26:25 2022] GET /identity/v3/roles/admin => generated 83 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:25.820030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8536740-1c7b-4d81-b865-fe72667e75e1 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:25.851319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4de88de-58a7-488e-8f97-bff1b215e7fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:25.851951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4de88de-58a7-488e-8f97-bff1b215e7fa None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:25.852238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4de88de-58a7-488e-8f97-bff1b215e7fa None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:25.852571 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4de88de-58a7-488e-8f97-bff1b215e7fa None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:25.853196 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4de88de-58a7-488e-8f97-bff1b215e7fa None admin] RBAC: Adding query filter params (name=member) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:25.853502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4de88de-58a7-488e-8f97-bff1b215e7fa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:25.855102 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4de88de-58a7-488e-8f97-bff1b215e7fa None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:25.868648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 58/115] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:25 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:25.868648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be453d0-6bef-40a0-aba8-9f70a7c4b8cc None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:25.868648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be453d0-6bef-40a0-aba8-9f70a7c4b8cc None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:25.868648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be453d0-6bef-40a0-aba8-9f70a7c4b8cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:25.884884 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:25.885479 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-4be453d0-6bef-40a0-aba8-9f70a7c4b8cc None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:25.886705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-4be453d0-6bef-40a0-aba8-9f70a7c4b8cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:25.942857 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8536740-1c7b-4d81-b865-fe72667e75e1 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:25.965965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 59/116] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:25.970728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf6eff0-c77b-4fb9-981c-37e472b815ea None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:25.970728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf6eff0-c77b-4fb9-981c-37e472b815ea None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:25.970728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf6eff0-c77b-4fb9-981c-37e472b815ea None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:25.970728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 60/117] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:25.971042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf6eff0-c77b-4fb9-981c-37e472b815ea None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:25.978889 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60a69cb-bc19-4b5d-a2d2-6d16041f562b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:25.979525 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60a69cb-bc19-4b5d-a2d2-6d16041f562b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:25.979806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60a69cb-bc19-4b5d-a2d2-6d16041f562b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:25.980076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60a69cb-bc19-4b5d-a2d2-6d16041f562b None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:25.981202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60a69cb-bc19-4b5d-a2d2-6d16041f562b None admin] RBAC: Authorizing `identity:get_project(project_id=333c93c6a3ef48b6806ea0d5f14ce211)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:25.982682 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60a69cb-bc19-4b5d-a2d2-6d16041f562b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:25.984276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 58/118] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:25 2022] GET /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211 => generated 298 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:25.989828 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60a69cb-bc19-4b5d-a2d2-6d16041f562b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:25.990289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:26.115295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60a69cb-bc19-4b5d-a2d2-6d16041f562b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:26.117397 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf6eff0-c77b-4fb9-981c-37e472b815ea None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:26.133628 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:26.150746 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c031c1-e802-4a73-9ba2-d47fefa04c7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:26.151416 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c031c1-e802-4a73-9ba2-d47fefa04c7a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:26.151772 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c031c1-e802-4a73-9ba2-d47fefa04c7a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:26.152120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c031c1-e802-4a73-9ba2-d47fefa04c7a None admin] PATH_INFO: `/v3/users/dd33c1df1ffc427888650d2537b8f6d7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:26.153327 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c031c1-e802-4a73-9ba2-d47fefa04c7a None admin] RBAC: Authorizing `identity:get_user(user_id=dd33c1df1ffc427888650d2537b8f6d7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:26.154897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c031c1-e802-4a73-9ba2-d47fefa04c7a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:26.156595 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 59/119] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:25 2022] GET /identity/v3/users/dd33c1df1ffc427888650d2537b8f6d7 => generated 275 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:26.157679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f135a9-b79c-4b7a-b1ef-bf2035266f49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:26.157965 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce94a2a3-7b60-4acb-be37-215215143784 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:26.158332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce94a2a3-7b60-4acb-be37-215215143784 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:26.158701 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce94a2a3-7b60-4acb-be37-215215143784 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:26.158986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f135a9-b79c-4b7a-b1ef-bf2035266f49 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:26.159408 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f135a9-b79c-4b7a-b1ef-bf2035266f49 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:26.159824 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f135a9-b79c-4b7a-b1ef-bf2035266f49 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:26.160800 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f135a9-b79c-4b7a-b1ef-bf2035266f49 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:26.161355 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f135a9-b79c-4b7a-b1ef-bf2035266f49 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:26.163132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f135a9-b79c-4b7a-b1ef-bf2035266f49 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:26.174558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 61/120] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:25 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:26.175652 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f135a9-b79c-4b7a-b1ef-bf2035266f49 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:26.188495 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ce94a2a3-7b60-4acb-be37-215215143784 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:26.190262 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ce94a2a3-7b60-4acb-be37-215215143784 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:26.259245 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:26.267197 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 60/121] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:26.268047 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce94a2a3-7b60-4acb-be37-215215143784 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:26.294248 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f135a9-b79c-4b7a-b1ef-bf2035266f49 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:26.342051 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade70bb1-2e9c-4d50-a078-f041ac4bb5e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:26.342998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade70bb1-2e9c-4d50-a078-f041ac4bb5e1 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:26.343286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade70bb1-2e9c-4d50-a078-f041ac4bb5e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:26.343568 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade70bb1-2e9c-4d50-a078-f041ac4bb5e1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:26.344573 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade70bb1-2e9c-4d50-a078-f041ac4bb5e1 None admin] RBAC: Adding query filter params (role.id=05a780eae85c41adab97f133b17a6ed4, scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211, user.id=41f973abcdc243e1b159643b7953c768) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:26.345757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade70bb1-2e9c-4d50-a078-f041ac4bb5e1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:26.347262 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade70bb1-2e9c-4d50-a078-f041ac4bb5e1 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:26.354249 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 62/122] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:26:26 2022] GET /identity/v3/role_assignments?user.id=41f973abcdc243e1b159643b7953c768&scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211&role.id=05a780eae85c41adab97f133b17a6ed4 => generated 259 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:26.354898 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade70bb1-2e9c-4d50-a078-f041ac4bb5e1 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:26.424523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce94a2a3-7b60-4acb-be37-215215143784 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:26.478213 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade70bb1-2e9c-4d50-a078-f041ac4bb5e1 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:26.483508 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:26.525556 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70b66be-9151-444a-be78-d0ccbe8a2497 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:26.529433 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70b66be-9151-444a-be78-d0ccbe8a2497 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:26.529721 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70b66be-9151-444a-be78-d0ccbe8a2497 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:26.530001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70b66be-9151-444a-be78-d0ccbe8a2497 None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:26.531295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70b66be-9151-444a-be78-d0ccbe8a2497 None admin] RBAC: Authorizing `identity:get_project(project_id=333c93c6a3ef48b6806ea0d5f14ce211)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:26.533524 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70b66be-9151-444a-be78-d0ccbe8a2497 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:26.535658 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 61/123] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:26 2022] GET /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211 => generated 298 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:26.536306 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70b66be-9151-444a-be78-d0ccbe8a2497 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:26.569548 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f63b28-ea3a-46c8-8f10-6a1270d68032 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:26.570381 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f63b28-ea3a-46c8-8f10-6a1270d68032 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:26.570724 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f63b28-ea3a-46c8-8f10-6a1270d68032 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:26.571154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f63b28-ea3a-46c8-8f10-6a1270d68032 None admin] PATH_INFO: `/v3/users/767ee8fd8728440799a75a0b00b134c0` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:26.572805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f63b28-ea3a-46c8-8f10-6a1270d68032 None admin] RBAC: Authorizing `identity:get_user(user_id=767ee8fd8728440799a75a0b00b134c0)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:26.574288 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f63b28-ea3a-46c8-8f10-6a1270d68032 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:26.576313 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 63/124] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:26 2022] GET /identity/v3/users/767ee8fd8728440799a75a0b00b134c0 => generated 247 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:26.577184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f63b28-ea3a-46c8-8f10-6a1270d68032 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:26.587302 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:26.658235 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70b66be-9151-444a-be78-d0ccbe8a2497 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:26.701931 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b38319-9196-48e3-ae66-3f732281b4a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:26.702493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b38319-9196-48e3-ae66-3f732281b4a4 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:26.702825 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b38319-9196-48e3-ae66-3f732281b4a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:26.703103 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b38319-9196-48e3-ae66-3f732281b4a4 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:26.703329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f63b28-ea3a-46c8-8f10-6a1270d68032 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:26.721468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b38319-9196-48e3-ae66-3f732281b4a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:26.728731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-d5b38319-9196-48e3-ae66-3f732281b4a4 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:26:26.728731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:26.728731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:26.728731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:26.728731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:26.728731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:26.728731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:26.728731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:26.728731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:26.728731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:26.728731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:26.728731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:26.728731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:26.728731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:26.728731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:26.728731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:26.729602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:26.729602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:26.729602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:26.729602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:26.729602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:26.729602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:26.729602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:26.729602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:26.729602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:26.729602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:26.729602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:26.729602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:26.729602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:26.729602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:26.729602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:26.729602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:26.730383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:26.730383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:26.730383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:26.730383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:26.730383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:26.730383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:26.730383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:26.730383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:26.730383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:26.730383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:26.730383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:26:26.730383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:26:26.731000 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 62/125] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:26:26 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:26.731550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b38319-9196-48e3-ae66-3f732281b4a4 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:26.751338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e90d2a8-5412-4706-9225-96cb76798377 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:26.752645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e90d2a8-5412-4706-9225-96cb76798377 None admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:26.753126 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e90d2a8-5412-4706-9225-96cb76798377 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:26.754057 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e90d2a8-5412-4706-9225-96cb76798377 None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211/users/dd33c1df1ffc427888650d2537b8f6d7/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:26.756737 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e90d2a8-5412-4706-9225-96cb76798377 None admin] RBAC: Authorizing `identity:create_grant(project_id=333c93c6a3ef48b6806ea0d5f14ce211, user_id=dd33c1df1ffc427888650d2537b8f6d7, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:26.759034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e90d2a8-5412-4706-9225-96cb76798377 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:26.786149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:26.842626 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 64/126] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:26:26 2022] PUT /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211/users/dd33c1df1ffc427888650d2537b8f6d7/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:26:26.843928 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e90d2a8-5412-4706-9225-96cb76798377 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:26.844413 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b38319-9196-48e3-ae66-3f732281b4a4 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:26.914537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84211cea-46fd-4d53-8920-f11328d73d5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:26.916033 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84211cea-46fd-4d53-8920-f11328d73d5a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:26.916533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84211cea-46fd-4d53-8920-f11328d73d5a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:26.917116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84211cea-46fd-4d53-8920-f11328d73d5a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:26.929780 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84211cea-46fd-4d53-8920-f11328d73d5a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:26.935843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-84211cea-46fd-4d53-8920-f11328d73d5a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:26.935843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:26.935843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:26.935843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:26.935843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:26.935843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:26.935843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:26.935843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:26.935843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:26.935843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:26.935843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:26.935843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:26.935843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:26.935843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:26.935843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:26.935843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:26.936861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:26.936861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:26.936861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:26.936861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:26.936861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:26.936861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:26.936861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:26.936861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:26.936861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:26.936861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:26.936861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:26.936861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:26.936861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:26.936861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:26.936861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:26.936861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:26.937670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:26.937670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:26.937670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:26.937670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:26.937670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:26.937670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:26.937670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:26.937670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:26.937670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:26.937670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:26.937670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:26.937670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:26:26.940201 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 63/127] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:26.941420 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84211cea-46fd-4d53-8920-f11328d73d5a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:27.027170 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:27.064882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84211cea-46fd-4d53-8920-f11328d73d5a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:27.067063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e90d2a8-5412-4706-9225-96cb76798377 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:27.070415 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:27.107655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e0b6ee-5e38-4d4f-bbdd-0d7c3f6cc40d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:27.108830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e0b6ee-5e38-4d4f-bbdd-0d7c3f6cc40d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:27.109297 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e0b6ee-5e38-4d4f-bbdd-0d7c3f6cc40d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:27.109721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e3465e-5159-4303-8425-03e9289dacb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:27.110035 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e0b6ee-5e38-4d4f-bbdd-0d7c3f6cc40d None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:27.110359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e3465e-5159-4303-8425-03e9289dacb5 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:27.110736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e3465e-5159-4303-8425-03e9289dacb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:27.111015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e3465e-5159-4303-8425-03e9289dacb5 None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:27.111564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e0b6ee-5e38-4d4f-bbdd-0d7c3f6cc40d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:27.112046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e0b6ee-5e38-4d4f-bbdd-0d7c3f6cc40d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:27.112156 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e3465e-5159-4303-8425-03e9289dacb5 None admin] RBAC: Authorizing `identity:get_project(project_id=333c93c6a3ef48b6806ea0d5f14ce211)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:27.113593 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e3465e-5159-4303-8425-03e9289dacb5 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:27.113917 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e0b6ee-5e38-4d4f-bbdd-0d7c3f6cc40d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:27.115416 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 65/128] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:26 2022] GET /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211 => generated 298 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:27.116221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e3465e-5159-4303-8425-03e9289dacb5 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:27.131561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 64/129] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:26:26 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:27.134980 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e0b6ee-5e38-4d4f-bbdd-0d7c3f6cc40d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:27.325214 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e3465e-5159-4303-8425-03e9289dacb5 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:27.346290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e0b6ee-5e38-4d4f-bbdd-0d7c3f6cc40d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:27.369834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acd5d9f-deba-4e30-a369-66da0f082204 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:27.370474 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acd5d9f-deba-4e30-a369-66da0f082204 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:27.370863 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acd5d9f-deba-4e30-a369-66da0f082204 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:27.371153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acd5d9f-deba-4e30-a369-66da0f082204 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:27.372018 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acd5d9f-deba-4e30-a369-66da0f082204 None admin] RBAC: Adding query filter params (name=member) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:27.372338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acd5d9f-deba-4e30-a369-66da0f082204 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:27.374142 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acd5d9f-deba-4e30-a369-66da0f082204 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:27.375934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:27.385527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 66/130] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:27 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:27.394928 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acd5d9f-deba-4e30-a369-66da0f082204 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:27.403285 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587a1972-91a0-4bee-94f0-ae2b76433c5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:27.404317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587a1972-91a0-4bee-94f0-ae2b76433c5b None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:27.404726 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587a1972-91a0-4bee-94f0-ae2b76433c5b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:27.405194 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587a1972-91a0-4bee-94f0-ae2b76433c5b None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211/users/767ee8fd8728440799a75a0b00b134c0/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:27.407929 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587a1972-91a0-4bee-94f0-ae2b76433c5b None admin] RBAC: Authorizing `identity:create_grant(project_id=333c93c6a3ef48b6806ea0d5f14ce211, user_id=767ee8fd8728440799a75a0b00b134c0, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:27.409703 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587a1972-91a0-4bee-94f0-ae2b76433c5b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:27.504420 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 65/131] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:26:27 2022] PUT /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211/users/767ee8fd8728440799a75a0b00b134c0/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:26:27.505819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587a1972-91a0-4bee-94f0-ae2b76433c5b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:27.529910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acd5d9f-deba-4e30-a369-66da0f082204 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:27.561794 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:27.603162 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07ce2b6-c5ff-4c9d-b2d6-988996f93697 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:27.604171 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07ce2b6-c5ff-4c9d-b2d6-988996f93697 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:27.604648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07ce2b6-c5ff-4c9d-b2d6-988996f93697 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:27.605114 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07ce2b6-c5ff-4c9d-b2d6-988996f93697 None admin] PATH_INFO: `/v3/users/dd33c1df1ffc427888650d2537b8f6d7` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:27.607881 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07ce2b6-c5ff-4c9d-b2d6-988996f93697 None admin] RBAC: Authorizing `identity:get_user(user_id=dd33c1df1ffc427888650d2537b8f6d7)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:27.609407 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07ce2b6-c5ff-4c9d-b2d6-988996f93697 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:27.611837 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 67/132] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:27 2022] GET /identity/v3/users/dd33c1df1ffc427888650d2537b8f6d7 => generated 275 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:27.618764 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07ce2b6-c5ff-4c9d-b2d6-988996f93697 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:27.695671 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587a1972-91a0-4bee-94f0-ae2b76433c5b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:27.743034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:27.771886 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c645100d-a73f-4e06-8579-5f40a2906c29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:27.772556 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c645100d-a73f-4e06-8579-5f40a2906c29 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:27.772846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c645100d-a73f-4e06-8579-5f40a2906c29 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:27.773120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c645100d-a73f-4e06-8579-5f40a2906c29 None admin] PATH_INFO: `/v3/users/dd33c1df1ffc427888650d2537b8f6d7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:27.776116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c645100d-a73f-4e06-8579-5f40a2906c29 None admin] RBAC: Authorizing `identity:get_user(user_id=dd33c1df1ffc427888650d2537b8f6d7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:27.777620 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c645100d-a73f-4e06-8579-5f40a2906c29 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:27.779463 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 66/133] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:27 2022] GET /identity/v3/users/dd33c1df1ffc427888650d2537b8f6d7 => generated 275 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:27.785323 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c645100d-a73f-4e06-8579-5f40a2906c29 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:27.838610 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07ce2b6-c5ff-4c9d-b2d6-988996f93697 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:27.883389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ac4601-fefb-478a-b481-14822e8eb002 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:27.883966 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ac4601-fefb-478a-b481-14822e8eb002 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:27.884254 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ac4601-fefb-478a-b481-14822e8eb002 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:27.884530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ac4601-fefb-478a-b481-14822e8eb002 None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:27.885672 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ac4601-fefb-478a-b481-14822e8eb002 None admin] RBAC: Authorizing `identity:get_project(project_id=333c93c6a3ef48b6806ea0d5f14ce211)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:27.887176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ac4601-fefb-478a-b481-14822e8eb002 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:27.889150 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 68/134] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:27 2022] GET /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211 => generated 298 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:27.895042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ac4601-fefb-478a-b481-14822e8eb002 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:27.923675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:27.932937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c645100d-a73f-4e06-8579-5f40a2906c29 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:28.002072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011f32a8-7def-4832-ae8f-52d6bdc5e062 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:28.002748 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011f32a8-7def-4832-ae8f-52d6bdc5e062 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:28.003035 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011f32a8-7def-4832-ae8f-52d6bdc5e062 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:28.003317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011f32a8-7def-4832-ae8f-52d6bdc5e062 None admin] PATH_INFO: `/v3/projects/a8737ba920904c149557cbac6c1301c6` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:28.009523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011f32a8-7def-4832-ae8f-52d6bdc5e062 None admin] RBAC: Authorizing `identity:get_project(project_id=a8737ba920904c149557cbac6c1301c6)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:28.011029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011f32a8-7def-4832-ae8f-52d6bdc5e062 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:28.012711 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 67/135] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:27 2022] GET /identity/v3/projects/a8737ba920904c149557cbac6c1301c6 => generated 312 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:28.019569 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011f32a8-7def-4832-ae8f-52d6bdc5e062 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:28.090629 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ac4601-fefb-478a-b481-14822e8eb002 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:28.195178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011f32a8-7def-4832-ae8f-52d6bdc5e062 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:28.204832 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:28.215821 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c504e735-4a5d-4618-b030-6897ee8182fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:28.216504 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c504e735-4a5d-4618-b030-6897ee8182fb None admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:28.216792 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c504e735-4a5d-4618-b030-6897ee8182fb None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:28.218512 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c504e735-4a5d-4618-b030-6897ee8182fb None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211/users/dd33c1df1ffc427888650d2537b8f6d7/roles/da6797f7fa24429095283b460be5c792` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:28.222943 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c504e735-4a5d-4618-b030-6897ee8182fb None admin] RBAC: Authorizing `identity:create_grant(project_id=333c93c6a3ef48b6806ea0d5f14ce211, user_id=dd33c1df1ffc427888650d2537b8f6d7, role_id=da6797f7fa24429095283b460be5c792)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:28.226454 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c504e735-4a5d-4618-b030-6897ee8182fb None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:28.247699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f1ff99-c942-4658-90a3-33c4af39e52d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:28.248384 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f1ff99-c942-4658-90a3-33c4af39e52d None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:28.248680 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f1ff99-c942-4658-90a3-33c4af39e52d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:28.248963 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f1ff99-c942-4658-90a3-33c4af39e52d None admin] PATH_INFO: `/v3/projects/a8737ba920904c149557cbac6c1301c6/users/dd33c1df1ffc427888650d2537b8f6d7/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:28.251008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f1ff99-c942-4658-90a3-33c4af39e52d None admin] RBAC: Authorizing `identity:create_grant(project_id=a8737ba920904c149557cbac6c1301c6, user_id=dd33c1df1ffc427888650d2537b8f6d7, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:28.252509 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f1ff99-c942-4658-90a3-33c4af39e52d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:28.304480 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 69/136] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:26:27 2022] PUT /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211/users/dd33c1df1ffc427888650d2537b8f6d7/roles/da6797f7fa24429095283b460be5c792 => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:26:28.305465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1451c30-ea01-47e8-9de4-150594bb748c None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:28.305756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1451c30-ea01-47e8-9de4-150594bb748c None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:28.306043 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1451c30-ea01-47e8-9de4-150594bb748c None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:28.307644 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 70/137] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:28.316909 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0269b231-02f6-42e4-8e28-cd6c8552a155 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:28.317326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0269b231-02f6-42e4-8e28-cd6c8552a155 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:28.317618 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0269b231-02f6-42e4-8e28-cd6c8552a155 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:28.332826 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 68/138] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:26:28 2022] PUT /identity/v3/projects/a8737ba920904c149557cbac6c1301c6/users/dd33c1df1ffc427888650d2537b8f6d7/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:26:28.354924 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-0269b231-02f6-42e4-8e28-cd6c8552a155 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:28.356359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0269b231-02f6-42e4-8e28-cd6c8552a155 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:28.456275 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 71/139] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:26:28.802657 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f1ff99-c942-4658-90a3-33c4af39e52d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:28.835241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:28.934874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f1ff99-c942-4658-90a3-33c4af39e52d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:28.975197 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43da760-dd19-442a-bd5b-052ae0789fab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:28.975848 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43da760-dd19-442a-bd5b-052ae0789fab None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:28.976132 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43da760-dd19-442a-bd5b-052ae0789fab None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:28.976408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43da760-dd19-442a-bd5b-052ae0789fab None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:28.977514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43da760-dd19-442a-bd5b-052ae0789fab None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:28.978986 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43da760-dd19-442a-bd5b-052ae0789fab None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:28.980598 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 69/140] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:26:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:28.986335 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0269b231-02f6-42e4-8e28-cd6c8552a155 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:29.106955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0269b231-02f6-42e4-8e28-cd6c8552a155 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:29.131378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:29.143782 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1de3051-dae7-4d55-ac1f-7a8367ed4b53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:29.144402 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1de3051-dae7-4d55-ac1f-7a8367ed4b53 None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:29.144686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1de3051-dae7-4d55-ac1f-7a8367ed4b53 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:29.144961 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1de3051-dae7-4d55-ac1f-7a8367ed4b53 None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:29.148273 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1de3051-dae7-4d55-ac1f-7a8367ed4b53 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:29.148273 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1de3051-dae7-4d55-ac1f-7a8367ed4b53 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:29.265523 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 72/141] 10.222.0.39 () {60 vars in 1132 bytes} [Fri Sep 30 09:26:28 2022] POST /identity/v3/users => generated 283 bytes in 279 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:26:29.567679 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe12595-7277-42ac-a639-99bb7c8a2a19 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:29.568078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe12595-7277-42ac-a639-99bb7c8a2a19 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:29.568252 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe12595-7277-42ac-a639-99bb7c8a2a19 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:29.569631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 70/142] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:29.575460 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84af2cdb-9197-4671-b692-989d4d094c28 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:29.575755 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84af2cdb-9197-4671-b692-989d4d094c28 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:29.576029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84af2cdb-9197-4671-b692-989d4d094c28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:29.599323 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfa546a-bc56-48f2-857f-7103ece0e396 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:29.599623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfa546a-bc56-48f2-857f-7103ece0e396 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:29.599831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfa546a-bc56-48f2-857f-7103ece0e396 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:29.600962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-84af2cdb-9197-4671-b692-989d4d094c28 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:29.601191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 71/143] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:26:29.602630 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-84af2cdb-9197-4671-b692-989d4d094c28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:29.606481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26be12b-a24c-4c03-994d-cc737ae3e1fc None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:29.607032 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26be12b-a24c-4c03-994d-cc737ae3e1fc None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:29.607339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26be12b-a24c-4c03-994d-cc737ae3e1fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:29.640623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c26be12b-a24c-4c03-994d-cc737ae3e1fc None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:29.642367 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c26be12b-a24c-4c03-994d-cc737ae3e1fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:29.714650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 73/144] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:29.750764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 72/145] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:30.282215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84af2cdb-9197-4671-b692-989d4d094c28 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:30.295686 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26be12b-a24c-4c03-994d-cc737ae3e1fc None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:30.330541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:30.464571 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84af2cdb-9197-4671-b692-989d4d094c28 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:30.474306 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26be12b-a24c-4c03-994d-cc737ae3e1fc None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:30.477726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:30.517564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb570252-3cb8-40c7-b686-2151c9519260 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:30.518212 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb570252-3cb8-40c7-b686-2151c9519260 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:30.518496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb570252-3cb8-40c7-b686-2151c9519260 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:30.518952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb570252-3cb8-40c7-b686-2151c9519260 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:30.530323 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb570252-3cb8-40c7-b686-2151c9519260 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:30.535867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-eb570252-3cb8-40c7-b686-2151c9519260 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 30 09:26:30.535867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:30.535867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:30.535867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:30.535867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:30.535867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:30.535867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:30.535867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:30.535867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:30.535867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:30.535867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:30.535867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:30.535867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:30.535867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:30.535867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:30.535867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:30.536837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:30.536837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:30.536837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:30.536837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:30.536837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:30.536837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:30.536837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:30.536837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:30.536837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:30.536837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:30.536837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:30.536837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:30.536837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:30.536837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:30.536837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:30.536837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:30.537642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:30.537642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:30.537642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:30.537642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:30.537642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:30.537642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:30.537642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:30.537642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:30.537642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:30.537642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:30.537642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 30 09:26:30.537642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:26:30.537642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 73/146] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:30 2022] GET /identity/v3/roles/reader => generated 84 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:26:30.538886 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d24ecf-6330-41ea-8aa4-f32a4386bd56 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:30.543206 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d24ecf-6330-41ea-8aa4-f32a4386bd56 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:30.543527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d24ecf-6330-41ea-8aa4-f32a4386bd56 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:30.544893 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 74/147] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:30 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:30.546671 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d24ecf-6330-41ea-8aa4-f32a4386bd56 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:30.573934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcb573d-59c8-4a77-ab59-7b2c8492f2bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:30.574683 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcb573d-59c8-4a77-ab59-7b2c8492f2bc None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:30.574969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcb573d-59c8-4a77-ab59-7b2c8492f2bc None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:30.575243 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcb573d-59c8-4a77-ab59-7b2c8492f2bc None admin] PATH_INFO: `/v3/roles/member` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:30.623255 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dcb573d-59c8-4a77-ab59-7b2c8492f2bc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:30.628834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-6dcb573d-59c8-4a77-ab59-7b2c8492f2bc None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:30.628834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:30.628834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:30.628834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:30.628834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:30.628834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:30.628834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:30.628834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:30.628834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:30.628834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:30.628834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:30.628834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:30.628834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:30.628834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:30.628834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:30.628834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:30.629742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:30.629742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:30.629742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:30.629742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:30.629742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:30.629742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:30.629742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:30.629742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:30.629742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:30.629742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:30.629742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:30.629742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:30.629742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:30.629742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:30.629742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:30.629742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:30.630527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:30.630527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:30.630527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:30.630527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:30.630527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:30.630527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:30.630527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:30.630527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:30.630527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:30.630527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:30.630527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:30.630527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:30.630527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 74/148] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:30 2022] GET /identity/v3/roles/member => generated 84 bytes in 348 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:30.631730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc802521-b527-40c6-95f5-5bb7ea186433 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:30.631730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc802521-b527-40c6-95f5-5bb7ea186433 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:30.631897 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc802521-b527-40c6-95f5-5bb7ea186433 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:30.641714 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:30.662974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d24ecf-6330-41ea-8aa4-f32a4386bd56 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:30.679619 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-fc802521-b527-40c6-95f5-5bb7ea186433 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:30.680904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-fc802521-b527-40c6-95f5-5bb7ea186433 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:30.699978 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21780fa1-ebff-45e7-b952-bbab52ef5657 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:30.700593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21780fa1-ebff-45e7-b952-bbab52ef5657 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:30.700871 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21780fa1-ebff-45e7-b952-bbab52ef5657 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:30.701142 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21780fa1-ebff-45e7-b952-bbab52ef5657 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:30.701804 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21780fa1-ebff-45e7-b952-bbab52ef5657 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:30.702105 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21780fa1-ebff-45e7-b952-bbab52ef5657 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:30.703530 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21780fa1-ebff-45e7-b952-bbab52ef5657 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:30.712781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 75/149] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:30 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:30.713717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21780fa1-ebff-45e7-b952-bbab52ef5657 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:30.792002 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 75/150] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:30.793103 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc802521-b527-40c6-95f5-5bb7ea186433 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:30.827116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21780fa1-ebff-45e7-b952-bbab52ef5657 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:30.863869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bfab27-b936-47b6-8ae7-10c936e16ad6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:30.864796 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bfab27-b936-47b6-8ae7-10c936e16ad6 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:30.865094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bfab27-b936-47b6-8ae7-10c936e16ad6 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:30.865502 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bfab27-b936-47b6-8ae7-10c936e16ad6 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:30.866288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8bfab27-b936-47b6-8ae7-10c936e16ad6 None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:30.866614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8bfab27-b936-47b6-8ae7-10c936e16ad6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:30.868008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8bfab27-b936-47b6-8ae7-10c936e16ad6 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:30.878110 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 76/151] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:30 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:30.884559 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bfab27-b936-47b6-8ae7-10c936e16ad6 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:30.888184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:30.913698 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:30.920071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc802521-b527-40c6-95f5-5bb7ea186433 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:30.963728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829cdee5-2c33-47e2-9d19-8e4eec35eb04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:30.964794 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829cdee5-2c33-47e2-9d19-8e4eec35eb04 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:30.965276 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829cdee5-2c33-47e2-9d19-8e4eec35eb04 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:30.965760 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829cdee5-2c33-47e2-9d19-8e4eec35eb04 None admin] PATH_INFO: `/v3/users/41f973abcdc243e1b159643b7953c768` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:30.967154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829cdee5-2c33-47e2-9d19-8e4eec35eb04 None admin] RBAC: Authorizing `identity:get_user(user_id=41f973abcdc243e1b159643b7953c768)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:30.968933 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829cdee5-2c33-47e2-9d19-8e4eec35eb04 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:30.971596 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 76/152] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:30 2022] GET /identity/v3/users/41f973abcdc243e1b159643b7953c768 => generated 289 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:30.986170 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829cdee5-2c33-47e2-9d19-8e4eec35eb04 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:31.001942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bfab27-b936-47b6-8ae7-10c936e16ad6 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:31.044609 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bd78b4-0a65-4652-9061-a1204bddf35d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:31.045222 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bd78b4-0a65-4652-9061-a1204bddf35d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:31.045473 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bd78b4-0a65-4652-9061-a1204bddf35d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:31.045900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bd78b4-0a65-4652-9061-a1204bddf35d None admin] PATH_INFO: `/v3/users/dd33c1df1ffc427888650d2537b8f6d7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:31.047546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bd78b4-0a65-4652-9061-a1204bddf35d None admin] RBAC: Authorizing `identity:get_user(user_id=dd33c1df1ffc427888650d2537b8f6d7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:31.049587 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bd78b4-0a65-4652-9061-a1204bddf35d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:31.052527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 77/153] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:30 2022] GET /identity/v3/users/dd33c1df1ffc427888650d2537b8f6d7 => generated 275 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:31.062973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bd78b4-0a65-4652-9061-a1204bddf35d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:31.143802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829cdee5-2c33-47e2-9d19-8e4eec35eb04 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:31.200348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:31.209267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa49dd3-2886-4669-b294-0bb332e92d74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:31.210263 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa49dd3-2886-4669-b294-0bb332e92d74 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:31.220393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa49dd3-2886-4669-b294-0bb332e92d74 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:31.220393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa49dd3-2886-4669-b294-0bb332e92d74 None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:31.220393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa49dd3-2886-4669-b294-0bb332e92d74 None admin] RBAC: Authorizing `identity:get_project(project_id=333c93c6a3ef48b6806ea0d5f14ce211)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:31.220393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa49dd3-2886-4669-b294-0bb332e92d74 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:31.221037 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 77/154] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:30 2022] GET /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211 => generated 298 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:31.223347 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa49dd3-2886-4669-b294-0bb332e92d74 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:31.266800 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:31.288785 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bd78b4-0a65-4652-9061-a1204bddf35d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:31.345599 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa49dd3-2886-4669-b294-0bb332e92d74 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:31.373171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85129c51-08e6-4cc8-9cb1-290c62ff4ea1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:31.373814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85129c51-08e6-4cc8-9cb1-290c62ff4ea1 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:31.374074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85129c51-08e6-4cc8-9cb1-290c62ff4ea1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:31.374451 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85129c51-08e6-4cc8-9cb1-290c62ff4ea1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:31.389333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3071bc53-66a9-4116-bf3e-e214eee2b151 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:31.390338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3071bc53-66a9-4116-bf3e-e214eee2b151 None admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:31.390899 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3071bc53-66a9-4116-bf3e-e214eee2b151 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:31.391398 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3071bc53-66a9-4116-bf3e-e214eee2b151 None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211/users/41f973abcdc243e1b159643b7953c768/roles/05a780eae85c41adab97f133b17a6ed4` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:31.393829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3071bc53-66a9-4116-bf3e-e214eee2b151 None admin] RBAC: Authorizing `identity:create_grant(project_id=333c93c6a3ef48b6806ea0d5f14ce211, user_id=41f973abcdc243e1b159643b7953c768, role_id=05a780eae85c41adab97f133b17a6ed4)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:31.395626 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3071bc53-66a9-4116-bf3e-e214eee2b151 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:31.463117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85129c51-08e6-4cc8-9cb1-290c62ff4ea1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:31.465089 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85129c51-08e6-4cc8-9cb1-290c62ff4ea1 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:31.480859 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 78/155] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:26:31 2022] PUT /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211/users/41f973abcdc243e1b159643b7953c768/roles/05a780eae85c41adab97f133b17a6ed4 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:26:31.484881 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35984d24-42bb-41a2-aad9-17bee31e4585 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:31.486072 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35984d24-42bb-41a2-aad9-17bee31e4585 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:31.486562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35984d24-42bb-41a2-aad9-17bee31e4585 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:31.489654 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 79/156] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:31 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:31.493607 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35984d24-42bb-41a2-aad9-17bee31e4585 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:31.631623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:31.697583 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 78/157] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 635 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:26:31.703075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37854908-b2e2-45d9-b57a-aaaa8df85910 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:31.704492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37854908-b2e2-45d9-b57a-aaaa8df85910 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:31.705973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37854908-b2e2-45d9-b57a-aaaa8df85910 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:31.710437 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:31.727321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35984d24-42bb-41a2-aad9-17bee31e4585 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:31.738372 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-37854908-b2e2-45d9-b57a-aaaa8df85910 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:31.740130 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-37854908-b2e2-45d9-b57a-aaaa8df85910 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:31.832695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 79/158] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:31.834007 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37854908-b2e2-45d9-b57a-aaaa8df85910 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:31.834516 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a12594f-10f6-4b6f-aa35-237ca98db14d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:31.835203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a12594f-10f6-4b6f-aa35-237ca98db14d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:31.835485 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a12594f-10f6-4b6f-aa35-237ca98db14d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:31.835762 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a12594f-10f6-4b6f-aa35-237ca98db14d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:31.867568 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a12594f-10f6-4b6f-aa35-237ca98db14d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:31.872558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-6a12594f-10f6-4b6f-aa35-237ca98db14d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:26:31.872558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:31.872558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:31.872558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:31.872558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:31.872558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:31.872558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:31.872558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:31.872558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:31.872558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:31.872558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:31.872558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:31.872558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:31.872558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:31.872558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:31.872558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:31.873502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:31.873502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:31.873502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:31.873502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:31.873502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:31.873502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:31.873502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:31.873502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:31.873502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:31.873502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:31.873502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:31.873502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:31.873502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:31.873502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:31.873502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:31.873502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:31.874958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:31.874958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:31.874958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:31.874958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:31.874958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:31.874958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:31.874958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:31.874958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:31.874958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:31.874958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:31.874958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:26:31.874958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:31.874958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 80/159] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:26:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 380 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:31.879519 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a12594f-10f6-4b6f-aa35-237ca98db14d None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:31.982474 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37854908-b2e2-45d9-b57a-aaaa8df85910 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:31.984929 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:32.046326 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6018f72-7d47-449e-8744-859dd0c288b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:32.047119 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6018f72-7d47-449e-8744-859dd0c288b0 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:32.047403 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6018f72-7d47-449e-8744-859dd0c288b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:32.047681 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6018f72-7d47-449e-8744-859dd0c288b0 None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:32.049893 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6018f72-7d47-449e-8744-859dd0c288b0 None admin] RBAC: Authorizing `identity:get_project(project_id=333c93c6a3ef48b6806ea0d5f14ce211)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:32.051391 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6018f72-7d47-449e-8744-859dd0c288b0 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:32.055651 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 80/160] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:31 2022] GET /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211 => generated 298 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:32.061526 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6018f72-7d47-449e-8744-859dd0c288b0 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:32.152553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a12594f-10f6-4b6f-aa35-237ca98db14d None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:32.227928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6018f72-7d47-449e-8744-859dd0c288b0 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:32.255429 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1d4bdd-1351-4e29-8a38-ecc25dc7b000 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:32.255982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1d4bdd-1351-4e29-8a38-ecc25dc7b000 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:32.256264 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1d4bdd-1351-4e29-8a38-ecc25dc7b000 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:32.256558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1d4bdd-1351-4e29-8a38-ecc25dc7b000 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:32.257221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1d4bdd-1351-4e29-8a38-ecc25dc7b000 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:32.257781 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1d4bdd-1351-4e29-8a38-ecc25dc7b000 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:32.259221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1d4bdd-1351-4e29-8a38-ecc25dc7b000 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:32.277893 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 81/161] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 399 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:32.284047 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1d4bdd-1351-4e29-8a38-ecc25dc7b000 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:32.296591 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31960aa2-0594-444c-9aa3-b00943b34d46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:32.297225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31960aa2-0594-444c-9aa3-b00943b34d46 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:32.297508 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31960aa2-0594-444c-9aa3-b00943b34d46 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:32.298444 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31960aa2-0594-444c-9aa3-b00943b34d46 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:32.299297 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31960aa2-0594-444c-9aa3-b00943b34d46 None admin] RBAC: Adding query filter params (role.id=025f58735650469d9fa6faca3e9e7998, scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211, user.id=dd33c1df1ffc427888650d2537b8f6d7) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:32.299613 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31960aa2-0594-444c-9aa3-b00943b34d46 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:32.301125 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31960aa2-0594-444c-9aa3-b00943b34d46 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:32.309535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 81/162] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:26:32 2022] GET /identity/v3/role_assignments?user.id=dd33c1df1ffc427888650d2537b8f6d7&scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211&role.id=025f58735650469d9fa6faca3e9e7998 => generated 608 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:32.312712 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f92c1ba-afb5-4bd3-af93-e394172bcdcd None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:32.313304 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f92c1ba-afb5-4bd3-af93-e394172bcdcd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:32.313630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f92c1ba-afb5-4bd3-af93-e394172bcdcd None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:32.315869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 82/163] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:32 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:26:32.317390 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:32.322369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c868032d-34de-4d9e-b39a-be3038178d97 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:32.322675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c868032d-34de-4d9e-b39a-be3038178d97 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:32.322955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c868032d-34de-4d9e-b39a-be3038178d97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:32.358969 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c868032d-34de-4d9e-b39a-be3038178d97 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:32.360291 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c868032d-34de-4d9e-b39a-be3038178d97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:32.394708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:32.478127 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 83/164] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:26:32.485309 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c868032d-34de-4d9e-b39a-be3038178d97 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:32.532040 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1d4bdd-1351-4e29-8a38-ecc25dc7b000 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:32.620397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c868032d-34de-4d9e-b39a-be3038178d97 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:32.627332 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e402196d-385f-49ea-9b5e-2c13964fb324 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:32.627985 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e402196d-385f-49ea-9b5e-2c13964fb324 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:32.628262 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e402196d-385f-49ea-9b5e-2c13964fb324 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:32.628533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e402196d-385f-49ea-9b5e-2c13964fb324 None admin] PATH_INFO: `/v3/users/767ee8fd8728440799a75a0b00b134c0` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:32.629713 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e402196d-385f-49ea-9b5e-2c13964fb324 None admin] RBAC: Authorizing `identity:get_user(user_id=767ee8fd8728440799a75a0b00b134c0)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:32.631203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e402196d-385f-49ea-9b5e-2c13964fb324 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:32.633359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 82/165] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:32 2022] GET /identity/v3/users/767ee8fd8728440799a75a0b00b134c0 => generated 247 bytes in 350 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:32.638653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e402196d-385f-49ea-9b5e-2c13964fb324 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:32.677178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0bafe7-9736-48b4-833b-7dc732d730ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:32.677824 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0bafe7-9736-48b4-833b-7dc732d730ad None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:32.679189 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0bafe7-9736-48b4-833b-7dc732d730ad None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:32.679524 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0bafe7-9736-48b4-833b-7dc732d730ad None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:32.705622 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0bafe7-9736-48b4-833b-7dc732d730ad None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:32.718388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-cb0bafe7-9736-48b4-833b-7dc732d730ad None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:26:32.718388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:32.718388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:32.718388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:32.718388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:32.718388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:32.718388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:32.718388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:32.718388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:32.718388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:32.718388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:32.718388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:32.718388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:32.718388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:32.718388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:32.718388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:32.719389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:32.719389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:32.719389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:32.719389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:32.719389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:32.719389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:32.719389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:32.719389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:32.719389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:32.719389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:32.719389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:32.719389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:32.719389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:32.719389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:32.719389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:32.719389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:32.720423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:32.720423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:32.720423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:32.720423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:32.720423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:32.720423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:32.720423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:32.720423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:32.720423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:32.720423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:32.720423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:26:32.720423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:26:32.720423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 84/166] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:26:32 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:32.726861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0bafe7-9736-48b4-833b-7dc732d730ad None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:32.782532 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:32.819519 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:32.837316 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e402196d-385f-49ea-9b5e-2c13964fb324 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:32.877442 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c13e5b9-b329-4b0f-baa0-1ca6f871d93b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:32.878109 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c13e5b9-b329-4b0f-baa0-1ca6f871d93b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:32.878394 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c13e5b9-b329-4b0f-baa0-1ca6f871d93b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:32.878730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c13e5b9-b329-4b0f-baa0-1ca6f871d93b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:32.952494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0bafe7-9736-48b4-833b-7dc732d730ad None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:32.952709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c13e5b9-b329-4b0f-baa0-1ca6f871d93b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:32.954119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c13e5b9-b329-4b0f-baa0-1ca6f871d93b None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:33.035403 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83603383-8ab3-476e-97be-6635dbdf0ce1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:33.035947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83603383-8ab3-476e-97be-6635dbdf0ce1 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:33.036225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83603383-8ab3-476e-97be-6635dbdf0ce1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:33.036494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83603383-8ab3-476e-97be-6635dbdf0ce1 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:33.037171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83603383-8ab3-476e-97be-6635dbdf0ce1 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:33.037475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83603383-8ab3-476e-97be-6635dbdf0ce1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:33.038912 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83603383-8ab3-476e-97be-6635dbdf0ce1 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:33.045749 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:33.052206 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 85/167] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:26:32 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 326 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:33.057793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83603383-8ab3-476e-97be-6635dbdf0ce1 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:33.066896 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 83/168] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 30 09:26:33.072786 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c13e5b9-b329-4b0f-baa0-1ca6f871d93b None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:33.182539 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c13e5b9-b329-4b0f-baa0-1ca6f871d93b None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:33.220172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f20638-8ff4-4562-b24d-afbf2c529182 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:33.220812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f20638-8ff4-4562-b24d-afbf2c529182 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:33.221096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f20638-8ff4-4562-b24d-afbf2c529182 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:33.221373 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f20638-8ff4-4562-b24d-afbf2c529182 None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:33.222560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f20638-8ff4-4562-b24d-afbf2c529182 None admin] RBAC: Authorizing `identity:get_project(project_id=333c93c6a3ef48b6806ea0d5f14ce211)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:33.224057 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f20638-8ff4-4562-b24d-afbf2c529182 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:33.225859 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 84/169] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:33 2022] GET /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211 => generated 298 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:33.231712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f20638-8ff4-4562-b24d-afbf2c529182 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:33.289241 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:33.304053 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:33.334918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83603383-8ab3-476e-97be-6635dbdf0ce1 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:33.342449 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f20638-8ff4-4562-b24d-afbf2c529182 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:33.381198 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9907492d-a683-46c0-a302-2343e5499324 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:33.381867 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9907492d-a683-46c0-a302-2343e5499324 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:33.382158 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9907492d-a683-46c0-a302-2343e5499324 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:33.383212 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9907492d-a683-46c0-a302-2343e5499324 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:33.383959 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9907492d-a683-46c0-a302-2343e5499324 None admin] RBAC: Adding query filter params (role.id=e874c311084c4c9fb7691bce461d319b, scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211, user.id=767ee8fd8728440799a75a0b00b134c0) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:33.384359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9907492d-a683-46c0-a302-2343e5499324 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:33.385737 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9907492d-a683-46c0-a302-2343e5499324 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:33.393285 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 85/170] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:26:33 2022] GET /identity/v3/role_assignments?user.id=767ee8fd8728440799a75a0b00b134c0&scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211&role.id=e874c311084c4c9fb7691bce461d319b => generated 608 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:33.398270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9907492d-a683-46c0-a302-2343e5499324 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:33.439980 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a04f48-adc7-425e-8757-8f987f556dc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:33.440657 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a04f48-adc7-425e-8757-8f987f556dc9 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:33.440925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a04f48-adc7-425e-8757-8f987f556dc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:33.441200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a04f48-adc7-425e-8757-8f987f556dc9 None admin] PATH_INFO: `/v3/users/dd33c1df1ffc427888650d2537b8f6d7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:33.442471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a04f48-adc7-425e-8757-8f987f556dc9 None admin] RBAC: Authorizing `identity:get_user(user_id=dd33c1df1ffc427888650d2537b8f6d7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:33.443985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a04f48-adc7-425e-8757-8f987f556dc9 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:33.445609 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 86/171] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:33 2022] GET /identity/v3/users/dd33c1df1ffc427888650d2537b8f6d7 => generated 275 bytes in 388 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:33.451175 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a04f48-adc7-425e-8757-8f987f556dc9 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:33.525547 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9907492d-a683-46c0-a302-2343e5499324 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:33.560594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:33.578863 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af27e4a-c995-4bde-8564-945e33655796 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:33.580127 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af27e4a-c995-4bde-8564-945e33655796 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:33.585041 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af27e4a-c995-4bde-8564-945e33655796 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:33.585041 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af27e4a-c995-4bde-8564-945e33655796 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:33.593829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af27e4a-c995-4bde-8564-945e33655796 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:33.605552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-9af27e4a-c995-4bde-8564-945e33655796 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:33.605552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:33.605552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:33.605552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:33.605552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:33.605552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:33.605552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:33.605552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:33.605552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:33.605552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:33.605552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:33.605552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:33.605552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:33.605552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:33.605552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:33.605552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:33.606604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:33.606604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:33.606604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:33.606604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:33.606604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:33.606604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:33.606604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:33.606604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:33.606604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:33.606604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:33.606604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:33.606604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:33.606604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:33.606604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:33.606604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:33.606604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:33.607488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:33.607488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:33.607488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:33.607488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:33.607488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:33.607488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:33.607488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:33.607488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:33.607488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:33.607488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:33.607488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:33.607488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:33.607488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 86/172] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:33.613954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af27e4a-c995-4bde-8564-945e33655796 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:33.743085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af27e4a-c995-4bde-8564-945e33655796 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:33.783331 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00accc69-09da-4f6d-929e-c4c38aae383c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:33.783954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00accc69-09da-4f6d-929e-c4c38aae383c None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:33.784236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00accc69-09da-4f6d-929e-c4c38aae383c None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:33.784558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00accc69-09da-4f6d-929e-c4c38aae383c None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:33.785236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00accc69-09da-4f6d-929e-c4c38aae383c None admin] RBAC: Adding query filter params (name=member) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:33.785544 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00accc69-09da-4f6d-929e-c4c38aae383c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:33.787133 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00accc69-09da-4f6d-929e-c4c38aae383c None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:33.799930 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 87/173] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:33 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:33.803044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00accc69-09da-4f6d-929e-c4c38aae383c None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:33.850266 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:33.864527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a04f48-adc7-425e-8757-8f987f556dc9 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:33.933717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35f4a22-b1c5-4911-9d21-85e20580ead6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:33.934379 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f4a22-b1c5-4911-9d21-85e20580ead6 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:33.934678 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f4a22-b1c5-4911-9d21-85e20580ead6 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:33.934948 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f4a22-b1c5-4911-9d21-85e20580ead6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:33.952156 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00accc69-09da-4f6d-929e-c4c38aae383c None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:33.979709 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:34.015561 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aaa9471-da76-46e2-b895-8ec55822711c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:34.016292 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aaa9471-da76-46e2-b895-8ec55822711c None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:34.017118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aaa9471-da76-46e2-b895-8ec55822711c None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:34.017395 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aaa9471-da76-46e2-b895-8ec55822711c None admin] PATH_INFO: `/v3/users/dd33c1df1ffc427888650d2537b8f6d7` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:34.019121 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aaa9471-da76-46e2-b895-8ec55822711c None admin] RBAC: Authorizing `identity:get_user(user_id=dd33c1df1ffc427888650d2537b8f6d7)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:34.020583 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aaa9471-da76-46e2-b895-8ec55822711c None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:34.022156 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 88/174] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:33 2022] GET /identity/v3/users/dd33c1df1ffc427888650d2537b8f6d7 => generated 275 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:34.024163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94800a4-05b4-40bd-85e9-d5fa717f5c47 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:34.024452 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94800a4-05b4-40bd-85e9-d5fa717f5c47 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:34.024722 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94800a4-05b4-40bd-85e9-d5fa717f5c47 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:34.025983 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 89/175] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:34.033062 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94800a4-05b4-40bd-85e9-d5fa717f5c47 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:34.064143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35f4a22-b1c5-4911-9d21-85e20580ead6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:34.065686 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35f4a22-b1c5-4911-9d21-85e20580ead6 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:34.231188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94800a4-05b4-40bd-85e9-d5fa717f5c47 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:34.235300 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 87/176] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 784 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:26:34.236543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937b6475-f60a-457a-aded-2d7452ad9fab None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:34.236824 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937b6475-f60a-457a-aded-2d7452ad9fab None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:34.237092 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937b6475-f60a-457a-aded-2d7452ad9fab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:34.258966 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:34.273181 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-937b6475-f60a-457a-aded-2d7452ad9fab None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:34.274379 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-937b6475-f60a-457a-aded-2d7452ad9fab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:34.296412 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb234893-3772-4686-9ba1-da7afef64e2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:34.297024 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb234893-3772-4686-9ba1-da7afef64e2e None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:34.297296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb234893-3772-4686-9ba1-da7afef64e2e None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:34.297561 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb234893-3772-4686-9ba1-da7afef64e2e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:34.368646 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:34.397467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 88/177] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:34.398323 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937b6475-f60a-457a-aded-2d7452ad9fab None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:34.429740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb234893-3772-4686-9ba1-da7afef64e2e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:34.436689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb234893-3772-4686-9ba1-da7afef64e2e None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:34.605584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937b6475-f60a-457a-aded-2d7452ad9fab None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:34.617782 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 90/178] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 586 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:26:34.623469 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb234893-3772-4686-9ba1-da7afef64e2e None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:34.650943 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:34.719346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efcc5214-733b-4eb5-8a5c-aab637724a10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:34.720016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcc5214-733b-4eb5-8a5c-aab637724a10 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:34.720293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcc5214-733b-4eb5-8a5c-aab637724a10 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:34.720565 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcc5214-733b-4eb5-8a5c-aab637724a10 None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:34.722739 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efcc5214-733b-4eb5-8a5c-aab637724a10 None admin] RBAC: Authorizing `identity:get_project(project_id=333c93c6a3ef48b6806ea0d5f14ce211)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:34.724196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efcc5214-733b-4eb5-8a5c-aab637724a10 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:34.725815 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 89/179] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:34 2022] GET /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211 => generated 298 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:34.726948 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caa4fa6-9141-4599-ac40-b10b1f839bdd None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:34.727224 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caa4fa6-9141-4599-ac40-b10b1f839bdd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:34.727490 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caa4fa6-9141-4599-ac40-b10b1f839bdd None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:34.728796 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 90/180] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:34 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:34.733749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5caa4fa6-9141-4599-ac40-b10b1f839bdd None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:34.757774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb234893-3772-4686-9ba1-da7afef64e2e None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:34.796605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833da562-df15-4605-9961-e1c17927909c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:34.797484 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833da562-df15-4605-9961-e1c17927909c None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:34.797845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833da562-df15-4605-9961-e1c17927909c None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:34.798524 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833da562-df15-4605-9961-e1c17927909c None admin] PATH_INFO: `/v3/projects/a8737ba920904c149557cbac6c1301c6` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:34.799972 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833da562-df15-4605-9961-e1c17927909c None admin] RBAC: Authorizing `identity:get_project(project_id=a8737ba920904c149557cbac6c1301c6)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:34.801449 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833da562-df15-4605-9961-e1c17927909c None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:34.804417 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 91/181] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:34 2022] GET /identity/v3/projects/a8737ba920904c149557cbac6c1301c6 => generated 312 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:34.805385 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d046e8f2-bae5-46c8-b33d-863c9d0822c2 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:34.805669 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d046e8f2-bae5-46c8-b33d-863c9d0822c2 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:34.805944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d046e8f2-bae5-46c8-b33d-863c9d0822c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:34.828690 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:34.832320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-d046e8f2-bae5-46c8-b33d-863c9d0822c2 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:34.833521 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d046e8f2-bae5-46c8-b33d-863c9d0822c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:34.880835 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5caa4fa6-9141-4599-ac40-b10b1f839bdd None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:34.917278 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8127e77-febb-4c43-bc54-4270d12c725e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:34.917883 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8127e77-febb-4c43-bc54-4270d12c725e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:34.918158 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8127e77-febb-4c43-bc54-4270d12c725e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:34.918426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8127e77-febb-4c43-bc54-4270d12c725e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:34.918626 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 92/182] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:34.919460 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8127e77-febb-4c43-bc54-4270d12c725e None admin] RBAC: Adding query filter params (role.id=da6797f7fa24429095283b460be5c792, scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211, user.id=dd33c1df1ffc427888650d2537b8f6d7) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:34.919565 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d046e8f2-bae5-46c8-b33d-863c9d0822c2 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:34.919765 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8127e77-febb-4c43-bc54-4270d12c725e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:34.921427 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8127e77-febb-4c43-bc54-4270d12c725e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:34.922875 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:34.929988 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 91/183] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:26:34 2022] GET /identity/v3/role_assignments?user.id=dd33c1df1ffc427888650d2537b8f6d7&scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211&role.id=da6797f7fa24429095283b460be5c792 => generated 608 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:34.939412 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8127e77-febb-4c43-bc54-4270d12c725e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:35.067604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d046e8f2-bae5-46c8-b33d-863c9d0822c2 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:35.067795 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8127e77-febb-4c43-bc54-4270d12c725e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:35.087618 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:35.112389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7765426-db92-445e-a5cd-5f006bef824d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:35.113551 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7765426-db92-445e-a5cd-5f006bef824d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:35.114389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7765426-db92-445e-a5cd-5f006bef824d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:35.114724 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7765426-db92-445e-a5cd-5f006bef824d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:35.115758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7765426-db92-445e-a5cd-5f006bef824d None admin] RBAC: Adding query filter params (role.id=025f58735650469d9fa6faca3e9e7998, scope.project.id=a8737ba920904c149557cbac6c1301c6, user.id=dd33c1df1ffc427888650d2537b8f6d7) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:35.116072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7765426-db92-445e-a5cd-5f006bef824d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:35.118833 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7765426-db92-445e-a5cd-5f006bef824d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:35.123228 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af61f81-2cdd-4d61-8c49-9b8013c3e0fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:35.123890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af61f81-2cdd-4d61-8c49-9b8013c3e0fe None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:35.124170 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af61f81-2cdd-4d61-8c49-9b8013c3e0fe None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:35.124445 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af61f81-2cdd-4d61-8c49-9b8013c3e0fe None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:35.129989 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 92/184] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:26:34 2022] GET /identity/v3/role_assignments?user.id=dd33c1df1ffc427888650d2537b8f6d7&scope.project.id=a8737ba920904c149557cbac6c1301c6&role.id=025f58735650469d9fa6faca3e9e7998 => generated 608 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:35.133222 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777262d2-ad42-4972-8544-b0d8c02d1f37 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:35.133452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777262d2-ad42-4972-8544-b0d8c02d1f37 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:35.134568 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777262d2-ad42-4972-8544-b0d8c02d1f37 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:35.138276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 93/185] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:35 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:35.146005 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a44671-b11f-404a-9fa5-1c22883a8ef5 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:35.146307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a44671-b11f-404a-9fa5-1c22883a8ef5 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:35.146542 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a44671-b11f-404a-9fa5-1c22883a8ef5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:35.150745 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af61f81-2cdd-4d61-8c49-9b8013c3e0fe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:35.161925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-2af61f81-2cdd-4d61-8c49-9b8013c3e0fe None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:26:35.161925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:35.161925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:35.161925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:35.161925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:35.161925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:35.161925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:35.161925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:35.161925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:35.161925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:35.161925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:35.161925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:35.161925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:35.161925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:35.161925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:35.161925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:35.164477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:35.164477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:35.164477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:35.164477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:35.164477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:35.164477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:35.164477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:35.164477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:35.164477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:35.164477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:35.164477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:35.164477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:35.164477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:35.164477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:35.164477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:35.164477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:35.166380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:35.166380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:35.166380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:35.166380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:35.166380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:35.166380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:35.166380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:35.166380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:35.166380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:35.166380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:35.166380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:26:35.166380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:35.166380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 93/186] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:26:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:35.170621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af61f81-2cdd-4d61-8c49-9b8013c3e0fe None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:35.187679 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-12a44671-b11f-404a-9fa5-1c22883a8ef5 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:35.191869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-12a44671-b11f-404a-9fa5-1c22883a8ef5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:35.325122 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:35.346259 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 94/187] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:35.347942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a44671-b11f-404a-9fa5-1c22883a8ef5 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:35.396613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af61f81-2cdd-4d61-8c49-9b8013c3e0fe None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:35.440044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e519e6aa-602f-4589-b459-f3d3d2cdeabb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:35.440656 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e519e6aa-602f-4589-b459-f3d3d2cdeabb None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:35.440929 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e519e6aa-602f-4589-b459-f3d3d2cdeabb None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:35.441202 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e519e6aa-602f-4589-b459-f3d3d2cdeabb None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:35.441924 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e519e6aa-602f-4589-b459-f3d3d2cdeabb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:35.442240 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e519e6aa-602f-4589-b459-f3d3d2cdeabb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:35.444078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e519e6aa-602f-4589-b459-f3d3d2cdeabb None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:35.453812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 94/188] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:35.454994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363eb74e-0a05-4a2c-bd79-74e567f5a645 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:35.455281 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363eb74e-0a05-4a2c-bd79-74e567f5a645 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:35.455551 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363eb74e-0a05-4a2c-bd79-74e567f5a645 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:35.456748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 95/189] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:35.462461 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b652387c-ed19-4369-8684-5bff8ef5bbf5 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:35.462792 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b652387c-ed19-4369-8684-5bff8ef5bbf5 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:35.463071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b652387c-ed19-4369-8684-5bff8ef5bbf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:35.483243 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-b652387c-ed19-4369-8684-5bff8ef5bbf5 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:35.484514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b652387c-ed19-4369-8684-5bff8ef5bbf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:35.501229 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a44671-b11f-404a-9fa5-1c22883a8ef5 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:35.513340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:35.546577 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdd37ea-d273-43f2-83d8-8e712c72abd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:35.548150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdd37ea-d273-43f2-83d8-8e712c72abd8 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:35.548575 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdd37ea-d273-43f2-83d8-8e712c72abd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:35.548987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdd37ea-d273-43f2-83d8-8e712c72abd8 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:35.564856 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbdd37ea-d273-43f2-83d8-8e712c72abd8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:35.570858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-dbdd37ea-d273-43f2-83d8-8e712c72abd8 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:26:35.570858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:35.570858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:35.570858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:35.570858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:35.570858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:35.570858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:35.570858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:35.570858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:35.570858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:35.570858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:35.570858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:35.570858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:35.570858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:35.570858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:35.570858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:35.571913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:35.571913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:35.571913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:35.571913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:35.571913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:35.571913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:35.571913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:35.571913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:35.571913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:35.571913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:35.571913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:35.571913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:35.571913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:35.571913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:35.571913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:35.571913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:35.572863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:35.572863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:35.572863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:35.572863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:35.572863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:35.572863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:35.572863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:35.572863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:35.572863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:35.572863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:35.572863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:26:35.572863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:26:35.574619 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 96/190] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:26:35.575036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 95/191] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:26:35 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:35.575458 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b652387c-ed19-4369-8684-5bff8ef5bbf5 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:35.584243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdd37ea-d273-43f2-83d8-8e712c72abd8 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:35.635491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:35.682943 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b652387c-ed19-4369-8684-5bff8ef5bbf5 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:35.692241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdd37ea-d273-43f2-83d8-8e712c72abd8 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:35.716985 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3be0af2-5242-44bf-b199-95a466d73575 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:35.717608 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3be0af2-5242-44bf-b199-95a466d73575 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:35.717892 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3be0af2-5242-44bf-b199-95a466d73575 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:35.718160 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3be0af2-5242-44bf-b199-95a466d73575 None admin] PATH_INFO: `/v3/users/a74eaf6245b34eee909a0d40c27a95b8` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:35.730858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1238abfb-fc34-435f-b32f-3da394888d76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:35.731553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3be0af2-5242-44bf-b199-95a466d73575 None admin] RBAC: Authorizing `identity:get_user(user_id=a74eaf6245b34eee909a0d40c27a95b8)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:35.731883 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1238abfb-fc34-435f-b32f-3da394888d76 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:35.732362 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1238abfb-fc34-435f-b32f-3da394888d76 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:35.732813 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1238abfb-fc34-435f-b32f-3da394888d76 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:35.733007 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3be0af2-5242-44bf-b199-95a466d73575 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:35.733822 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1238abfb-fc34-435f-b32f-3da394888d76 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:35.734301 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1238abfb-fc34-435f-b32f-3da394888d76 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:35.734608 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 97/192] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:35 2022] GET /identity/v3/users/a74eaf6245b34eee909a0d40c27a95b8 => generated 283 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:35.735946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3be0af2-5242-44bf-b199-95a466d73575 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:35.736916 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1238abfb-fc34-435f-b32f-3da394888d76 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:35.748418 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 96/193] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:26:35 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:35.751495 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1238abfb-fc34-435f-b32f-3da394888d76 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:35.782788 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:35.854702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1238abfb-fc34-435f-b32f-3da394888d76 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:35.855838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3be0af2-5242-44bf-b199-95a466d73575 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:35.871833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:35.889948 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632268f0-e7aa-4017-9994-f280ed5ea89d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:35.890726 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632268f0-e7aa-4017-9994-f280ed5ea89d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:35.891009 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632268f0-e7aa-4017-9994-f280ed5ea89d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:35.891302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632268f0-e7aa-4017-9994-f280ed5ea89d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:35.891555 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f20507-a7c6-4016-89ec-912fcf00774d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:35.892184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f20507-a7c6-4016-89ec-912fcf00774d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:35.892458 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f20507-a7c6-4016-89ec-912fcf00774d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:35.892725 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f20507-a7c6-4016-89ec-912fcf00774d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:35.893797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f20507-a7c6-4016-89ec-912fcf00774d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:35.895307 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f20507-a7c6-4016-89ec-912fcf00774d None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:35.897281 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 98/194] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:26:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:35.898527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f20507-a7c6-4016-89ec-912fcf00774d None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:35.958026 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632268f0-e7aa-4017-9994-f280ed5ea89d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:35.959431 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632268f0-e7aa-4017-9994-f280ed5ea89d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:35.988616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:35.996212 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f20507-a7c6-4016-89ec-912fcf00774d None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:36.028832 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97bb05b-96ed-4e1c-beb6-22520b7d8f13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:36.029454 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97bb05b-96ed-4e1c-beb6-22520b7d8f13 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:36.029727 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97bb05b-96ed-4e1c-beb6-22520b7d8f13 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:36.029996 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97bb05b-96ed-4e1c-beb6-22520b7d8f13 None admin] PATH_INFO: `/v3/users/a74eaf6245b34eee909a0d40c27a95b8` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:36.031136 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97bb05b-96ed-4e1c-beb6-22520b7d8f13 None admin] RBAC: Authorizing `identity:get_user(user_id=a74eaf6245b34eee909a0d40c27a95b8)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:36.032576 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97bb05b-96ed-4e1c-beb6-22520b7d8f13 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:36.034133 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 99/195] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:35 2022] GET /identity/v3/users/a74eaf6245b34eee909a0d40c27a95b8 => generated 283 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:36.034911 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97bb05b-96ed-4e1c-beb6-22520b7d8f13 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:36.065549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 97/196] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:26:36.065549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632268f0-e7aa-4017-9994-f280ed5ea89d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:36.082612 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:36.132923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97bb05b-96ed-4e1c-beb6-22520b7d8f13 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:36.164873 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438a1127-1495-4086-be37-c0f3e2e73551 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:36.165501 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438a1127-1495-4086-be37-c0f3e2e73551 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:36.165614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632268f0-e7aa-4017-9994-f280ed5ea89d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:36.165778 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438a1127-1495-4086-be37-c0f3e2e73551 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:36.166048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438a1127-1495-4086-be37-c0f3e2e73551 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:36.177604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438a1127-1495-4086-be37-c0f3e2e73551 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:36.183050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-438a1127-1495-4086-be37-c0f3e2e73551 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 30 09:26:36.183050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:36.183050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:36.183050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:36.183050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:36.183050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:36.183050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:36.183050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:36.183050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:36.183050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:36.183050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:36.183050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:36.183050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:36.183050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:36.183050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:36.183050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:36.184016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:36.184016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:36.184016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:36.184016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:36.184016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:36.184016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:36.184016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:36.184016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:36.184016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:36.184016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:36.184016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:36.184016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:36.184016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:36.184016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:36.184016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:36.184016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:36.184847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:36.184847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:36.184847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:36.184847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:36.184847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:36.184847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:36.184847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:36.184847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:36.184847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:36.184847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:36.184847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 30 09:26:36.184847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:36.184847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 100/197] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:36 2022] GET /identity/v3/roles/reader => generated 84 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:36.185544 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438a1127-1495-4086-be37-c0f3e2e73551 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:36.199245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:36.204514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cc50af-7a88-48b6-bd63-f293de48d92d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:36.205054 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cc50af-7a88-48b6-bd63-f293de48d92d None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:36.205382 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cc50af-7a88-48b6-bd63-f293de48d92d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:36.205778 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cc50af-7a88-48b6-bd63-f293de48d92d None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:36.206433 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3cc50af-7a88-48b6-bd63-f293de48d92d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:36.207923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3cc50af-7a88-48b6-bd63-f293de48d92d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:36.287207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438a1127-1495-4086-be37-c0f3e2e73551 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:36.296888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:36.319127 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a35509-e2a4-4ccc-a472-14d72f0f6385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:36.319758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a35509-e2a4-4ccc-a472-14d72f0f6385 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:36.320040 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a35509-e2a4-4ccc-a472-14d72f0f6385 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:36.320314 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a35509-e2a4-4ccc-a472-14d72f0f6385 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:36.383243 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a35509-e2a4-4ccc-a472-14d72f0f6385 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:36.384696 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a35509-e2a4-4ccc-a472-14d72f0f6385 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:36.478525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 101/198] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:26:36.479440 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a35509-e2a4-4ccc-a472-14d72f0f6385 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:36.497216 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 98/199] 10.222.0.39 () {60 vars in 1132 bytes} [Fri Sep 30 09:26:36 2022] POST /identity/v3/users => generated 297 bytes in 436 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:26:36.507311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cc50af-7a88-48b6-bd63-f293de48d92d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:36.530201 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:36.589541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a35509-e2a4-4ccc-a472-14d72f0f6385 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:36.622068 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cc50af-7a88-48b6-bd63-f293de48d92d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:36.629310 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f0e379-1859-46b6-9fc2-2bd2550b90c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:36.630498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f0e379-1859-46b6-9fc2-2bd2550b90c5 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:36.630813 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f0e379-1859-46b6-9fc2-2bd2550b90c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:36.631091 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f0e379-1859-46b6-9fc2-2bd2550b90c5 None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:36.632899 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f0e379-1859-46b6-9fc2-2bd2550b90c5 None admin] RBAC: Authorizing `identity:get_project(project_id=25655c9cb779428f96b5430e993f1337)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:36.634821 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f0e379-1859-46b6-9fc2-2bd2550b90c5 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:36.636588 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 102/200] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:36 2022] GET /identity/v3/projects/25655c9cb779428f96b5430e993f1337 => generated 302 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:36.637479 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f0e379-1859-46b6-9fc2-2bd2550b90c5 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:36.667259 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4642ec-8b22-4c10-ba20-d98c0755352e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:36.667487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4642ec-8b22-4c10-ba20-d98c0755352e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:36.667694 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4642ec-8b22-4c10-ba20-d98c0755352e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:36.667975 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4642ec-8b22-4c10-ba20-d98c0755352e None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:36.668658 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4642ec-8b22-4c10-ba20-d98c0755352e None admin] RBAC: Adding query filter params (name=reader) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:36.668968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4642ec-8b22-4c10-ba20-d98c0755352e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:36.670426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4642ec-8b22-4c10-ba20-d98c0755352e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:36.680193 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 99/201] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:36 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:36.683457 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4642ec-8b22-4c10-ba20-d98c0755352e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:36.730188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:36.744836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:36.759808 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f0e379-1859-46b6-9fc2-2bd2550b90c5 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:36.805348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3980c089-d96a-4805-a92d-ab42646f5619 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:36.806000 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3980c089-d96a-4805-a92d-ab42646f5619 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:36.806283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3980c089-d96a-4805-a92d-ab42646f5619 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:36.807144 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3980c089-d96a-4805-a92d-ab42646f5619 None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:36.808807 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3980c089-d96a-4805-a92d-ab42646f5619 None admin] RBAC: Authorizing `identity:get_project(project_id=25655c9cb779428f96b5430e993f1337)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:36.809585 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3980c089-d96a-4805-a92d-ab42646f5619 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:36.811162 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 103/202] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:36 2022] GET /identity/v3/projects/25655c9cb779428f96b5430e993f1337 => generated 302 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:36.811934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3980c089-d96a-4805-a92d-ab42646f5619 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:36.816536 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4642ec-8b22-4c10-ba20-d98c0755352e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:36.857229 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511cc2d2-0d10-4a82-82db-ac8b94db38c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:36.857768 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511cc2d2-0d10-4a82-82db-ac8b94db38c6 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:36.858051 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511cc2d2-0d10-4a82-82db-ac8b94db38c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:36.858322 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511cc2d2-0d10-4a82-82db-ac8b94db38c6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:36.859639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511cc2d2-0d10-4a82-82db-ac8b94db38c6 None admin] RBAC: Adding query filter params (role.id=e874c311084c4c9fb7691bce461d319b, scope.project.id=25655c9cb779428f96b5430e993f1337, user.id=a74eaf6245b34eee909a0d40c27a95b8) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:36.859639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511cc2d2-0d10-4a82-82db-ac8b94db38c6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:36.860893 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511cc2d2-0d10-4a82-82db-ac8b94db38c6 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:36.867387 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 100/203] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:26:36 2022] GET /identity/v3/role_assignments?user.id=a74eaf6245b34eee909a0d40c27a95b8&scope.project.id=25655c9cb779428f96b5430e993f1337&role.id=e874c311084c4c9fb7691bce461d319b => generated 259 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:36.868231 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511cc2d2-0d10-4a82-82db-ac8b94db38c6 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:36.948814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3980c089-d96a-4805-a92d-ab42646f5619 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:36.995047 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c0bfec-e9fd-4aab-ad8f-5eac8ea90259 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:36.995938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c0bfec-e9fd-4aab-ad8f-5eac8ea90259 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:36.996227 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c0bfec-e9fd-4aab-ad8f-5eac8ea90259 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:36.996502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c0bfec-e9fd-4aab-ad8f-5eac8ea90259 None admin] PATH_INFO: `/v3/users/41f973abcdc243e1b159643b7953c768` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:36.997305 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511cc2d2-0d10-4a82-82db-ac8b94db38c6 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:36.997650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c0bfec-e9fd-4aab-ad8f-5eac8ea90259 None admin] RBAC: Authorizing `identity:get_user(user_id=41f973abcdc243e1b159643b7953c768)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:36.999318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c0bfec-e9fd-4aab-ad8f-5eac8ea90259 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:37.002505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 104/204] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:36 2022] GET /identity/v3/users/41f973abcdc243e1b159643b7953c768 => generated 289 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:37.008854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c0bfec-e9fd-4aab-ad8f-5eac8ea90259 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:37.027010 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:37.050301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:37.060818 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0509968b-51d5-4ef0-a863-9c48099fd3ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:37.061554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0509968b-51d5-4ef0-a863-9c48099fd3ac None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:37.061845 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0509968b-51d5-4ef0-a863-9c48099fd3ac None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:37.062173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0509968b-51d5-4ef0-a863-9c48099fd3ac None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:37.062953 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0509968b-51d5-4ef0-a863-9c48099fd3ac None admin] RBAC: Adding query filter params (role.id=da6797f7fa24429095283b460be5c792, scope.project.id=25655c9cb779428f96b5430e993f1337, user.id=a74eaf6245b34eee909a0d40c27a95b8) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:37.063800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0509968b-51d5-4ef0-a863-9c48099fd3ac None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:37.066377 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0509968b-51d5-4ef0-a863-9c48099fd3ac None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:37.082185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 101/205] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:26:36 2022] GET /identity/v3/role_assignments?user.id=a74eaf6245b34eee909a0d40c27a95b8&scope.project.id=25655c9cb779428f96b5430e993f1337&role.id=da6797f7fa24429095283b460be5c792 => generated 259 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:37.187207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c0bfec-e9fd-4aab-ad8f-5eac8ea90259 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:37.239394 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7571999-4fb7-4d3f-9b6a-53ee476d6a3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:37.240010 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7571999-4fb7-4d3f-9b6a-53ee476d6a3d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:37.240348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7571999-4fb7-4d3f-9b6a-53ee476d6a3d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:37.240624 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7571999-4fb7-4d3f-9b6a-53ee476d6a3d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:37.326919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7571999-4fb7-4d3f-9b6a-53ee476d6a3d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:37.328528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7571999-4fb7-4d3f-9b6a-53ee476d6a3d None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:37.344814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:37.456239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 105/206] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 448 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:26:37.461735 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0509968b-51d5-4ef0-a863-9c48099fd3ac None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:37.564824 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0509968b-51d5-4ef0-a863-9c48099fd3ac None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:37.599107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e0407a-aed8-403a-a068-c25d234d1e47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:37.600059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e0407a-aed8-403a-a068-c25d234d1e47 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:37.600519 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e0407a-aed8-403a-a068-c25d234d1e47 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:37.600976 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e0407a-aed8-403a-a068-c25d234d1e47 None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:37.602303 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e0407a-aed8-403a-a068-c25d234d1e47 None admin] RBAC: Authorizing `identity:get_project(project_id=333c93c6a3ef48b6806ea0d5f14ce211)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:37.603985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e0407a-aed8-403a-a068-c25d234d1e47 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:37.606122 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 102/207] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:37 2022] GET /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211 => generated 298 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:37.612690 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7571999-4fb7-4d3f-9b6a-53ee476d6a3d None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:37.745591 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7571999-4fb7-4d3f-9b6a-53ee476d6a3d None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:37.760224 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:37.780837 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10695520-6324-4d7f-993b-0bac54830007 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:37.781457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10695520-6324-4d7f-993b-0bac54830007 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:37.782346 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10695520-6324-4d7f-993b-0bac54830007 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:37.782648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10695520-6324-4d7f-993b-0bac54830007 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:37.783384 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10695520-6324-4d7f-993b-0bac54830007 None admin] RBAC: Adding query filter params (role.id=05a780eae85c41adab97f133b17a6ed4, scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211, user.id=41f973abcdc243e1b159643b7953c768) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:37.783693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10695520-6324-4d7f-993b-0bac54830007 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:37.785170 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10695520-6324-4d7f-993b-0bac54830007 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:37.791940 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 106/208] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:26:37 2022] GET /identity/v3/role_assignments?user.id=41f973abcdc243e1b159643b7953c768&scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211&role.id=05a780eae85c41adab97f133b17a6ed4 => generated 608 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:39.048594 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83a4924-27e1-42a8-b68f-4502196802b0 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:39.049143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83a4924-27e1-42a8-b68f-4502196802b0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:39.049143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83a4924-27e1-42a8-b68f-4502196802b0 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:39.050622 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 103/209] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:39.056164 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c40112d-5edd-4069-a5fb-7a7e81e75024 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:39.056561 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c40112d-5edd-4069-a5fb-7a7e81e75024 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:39.056751 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c40112d-5edd-4069-a5fb-7a7e81e75024 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:39.076135 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-9c40112d-5edd-4069-a5fb-7a7e81e75024 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:39.077366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-9c40112d-5edd-4069-a5fb-7a7e81e75024 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:39.134464 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991fc197-ad58-4e66-80f2-7cc745b57c7f None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:39.134819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991fc197-ad58-4e66-80f2-7cc745b57c7f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:39.135107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991fc197-ad58-4e66-80f2-7cc745b57c7f None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:39.136440 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 104/210] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:26:39.141592 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3b5258-5262-4dcf-89b3-165a8d5a8b29 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:39.141866 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3b5258-5262-4dcf-89b3-165a8d5a8b29 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:39.142140 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3b5258-5262-4dcf-89b3-165a8d5a8b29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:39.151455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 107/211] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:39.164504 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-da3b5258-5262-4dcf-89b3-165a8d5a8b29 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:39.165955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-da3b5258-5262-4dcf-89b3-165a8d5a8b29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:39.224471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:39.253183 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 105/212] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:26:39.332697 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed0b9b0-4a9d-49e4-b4c0-a9f02946288f None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:39.332907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed0b9b0-4a9d-49e4-b4c0-a9f02946288f None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:39.333176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed0b9b0-4a9d-49e4-b4c0-a9f02946288f None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:39.334804 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 108/213] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:39.339972 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b08476-18cc-4f20-b626-9a11e4acf292 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:39.340261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b08476-18cc-4f20-b626-9a11e4acf292 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:39.340761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b08476-18cc-4f20-b626-9a11e4acf292 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:39.360278 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f3b08476-18cc-4f20-b626-9a11e4acf292 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:39.362375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f3b08476-18cc-4f20-b626-9a11e4acf292 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:39.430871 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 106/214] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:39.438895 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed0b9b0-4a9d-49e4-b4c0-a9f02946288f None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:39.542408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b08476-18cc-4f20-b626-9a11e4acf292 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:39.551000 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed0b9b0-4a9d-49e4-b4c0-a9f02946288f None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:39.561420 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:39.588912 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efe21c1-b261-4c31-8475-b2b01f272f73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:39.589975 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efe21c1-b261-4c31-8475-b2b01f272f73 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:39.590455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efe21c1-b261-4c31-8475-b2b01f272f73 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:39.591066 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efe21c1-b261-4c31-8475-b2b01f272f73 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:39.605486 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efe21c1-b261-4c31-8475-b2b01f272f73 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:39.613476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-1efe21c1-b261-4c31-8475-b2b01f272f73 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:39.613476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:39.613476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:39.613476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:39.613476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:39.613476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:39.613476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:39.613476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:39.613476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:39.613476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:39.613476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:39.613476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:39.613476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:39.613476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:39.613476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:39.613476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:39.614464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:39.614464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:39.614464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:39.614464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:39.614464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:39.614464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:39.614464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:39.614464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:39.614464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:39.614464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:39.614464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:39.614464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:39.614464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:39.614464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:39.614464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:39.614464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:39.615299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:39.615299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:39.615299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:39.615299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:39.615299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:39.615299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:39.615299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:39.615299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:39.615299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:39.615299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:39.615299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:39.615299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:39.615925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 109/215] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:39 2022] GET /identity/v3/roles/member => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:39.622391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efe21c1-b261-4c31-8475-b2b01f272f73 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:39.670004 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b08476-18cc-4f20-b626-9a11e4acf292 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:39.678582 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:39.707318 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c4eadf-1499-467b-a1d0-780069c3a9ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:39.707959 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c4eadf-1499-467b-a1d0-780069c3a9ea None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:39.708240 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c4eadf-1499-467b-a1d0-780069c3a9ea None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:39.708508 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c4eadf-1499-467b-a1d0-780069c3a9ea None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:39.709602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c4eadf-1499-467b-a1d0-780069c3a9ea None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:39.711165 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c4eadf-1499-467b-a1d0-780069c3a9ea None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:39.712731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 107/216] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:26:39 2022] GET /identity/v3/domains/default => generated 199 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:39.714749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c4eadf-1499-467b-a1d0-780069c3a9ea None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:39.740583 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efe21c1-b261-4c31-8475-b2b01f272f73 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:39.775326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5eeac38-10bd-487d-87c7-c959ad89ab65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:39.775940 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5eeac38-10bd-487d-87c7-c959ad89ab65 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:39.776222 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5eeac38-10bd-487d-87c7-c959ad89ab65 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:39.776495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5eeac38-10bd-487d-87c7-c959ad89ab65 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:39.777154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5eeac38-10bd-487d-87c7-c959ad89ab65 None admin] RBAC: Adding query filter params (name=member) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:39.777456 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5eeac38-10bd-487d-87c7-c959ad89ab65 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:39.778884 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5eeac38-10bd-487d-87c7-c959ad89ab65 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:39.786692 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 110/217] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:39 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:39.787465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5eeac38-10bd-487d-87c7-c959ad89ab65 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:39.816763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:39.820929 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c4eadf-1499-467b-a1d0-780069c3a9ea None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:39.854337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3033e6a2-8337-4422-bb87-2258c69d5e2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:39.854922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3033e6a2-8337-4422-bb87-2258c69d5e2f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:39.855206 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3033e6a2-8337-4422-bb87-2258c69d5e2f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:39.855474 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3033e6a2-8337-4422-bb87-2258c69d5e2f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:39.865938 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3033e6a2-8337-4422-bb87-2258c69d5e2f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:39.870810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-3033e6a2-8337-4422-bb87-2258c69d5e2f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:26:39.870810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:39.870810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:39.870810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:39.870810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:39.870810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:39.870810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:39.870810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:39.870810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:39.870810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:39.870810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:39.870810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:39.870810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:39.870810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:39.870810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:39.870810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:39.871832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:39.871832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:39.871832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:39.871832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:39.871832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:39.871832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:39.871832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:39.871832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:39.871832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:39.871832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:39.871832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:39.871832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:39.871832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:39.871832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:39.871832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:39.871832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:39.872661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:39.872661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:39.872661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:39.872661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:39.872661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:39.872661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:39.872661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:39.872661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:39.872661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:39.872661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:39.872661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:26:39.872661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:26:39.872661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 108/218] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:26:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:39.873519 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406d32cc-4de4-42af-a399-8ffff7840833 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:39.873519 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406d32cc-4de4-42af-a399-8ffff7840833 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:39.873519 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406d32cc-4de4-42af-a399-8ffff7840833 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:39.874808 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 109/219] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:39 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:39.875516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406d32cc-4de4-42af-a399-8ffff7840833 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:39.890789 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5eeac38-10bd-487d-87c7-c959ad89ab65 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:39.910644 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:39.924180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cfcccb-c3e2-4748-9689-0980571647a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:39.924781 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cfcccb-c3e2-4748-9689-0980571647a5 None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:39.925067 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cfcccb-c3e2-4748-9689-0980571647a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:39.925339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cfcccb-c3e2-4748-9689-0980571647a5 None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:39.925990 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cfcccb-c3e2-4748-9689-0980571647a5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:39.927478 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cfcccb-c3e2-4748-9689-0980571647a5 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:39.976944 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406d32cc-4de4-42af-a399-8ffff7840833 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:40.009397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45688fa3-0afa-45e0-bcdf-000f0eecd5f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:40.010017 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45688fa3-0afa-45e0-bcdf-000f0eecd5f6 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:40.010297 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45688fa3-0afa-45e0-bcdf-000f0eecd5f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:40.010568 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45688fa3-0afa-45e0-bcdf-000f0eecd5f6 None admin] PATH_INFO: `/v3/users/01598eff6b08454eab252b4d1d3de8dd` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:40.014735 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 111/220] 10.222.0.39 () {60 vars in 1132 bytes} [Fri Sep 30 09:26:39 2022] POST /identity/v3/users => generated 297 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:26:40.014735 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cfcccb-c3e2-4748-9689-0980571647a5 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:40.026933 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45688fa3-0afa-45e0-bcdf-000f0eecd5f6 None admin] RBAC: Authorizing `identity:get_user(user_id=01598eff6b08454eab252b4d1d3de8dd)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:40.026933 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45688fa3-0afa-45e0-bcdf-000f0eecd5f6 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:40.038818 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 110/221] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:39 2022] GET /identity/v3/users/01598eff6b08454eab252b4d1d3de8dd => generated 297 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:40.038818 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd81378-ab26-4614-ae20-3769d12bc8c5 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:40.038818 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd81378-ab26-4614-ae20-3769d12bc8c5 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:40.038818 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd81378-ab26-4614-ae20-3769d12bc8c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:40.050242 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-bdd81378-ab26-4614-ae20-3769d12bc8c5 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:40.051506 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-bdd81378-ab26-4614-ae20-3769d12bc8c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:40.142715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 111/222] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:40.142715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd81378-ab26-4614-ae20-3769d12bc8c5 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:40.147071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:40.169820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cfcccb-c3e2-4748-9689-0980571647a5 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:40.173441 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:40.234776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e7bfc8-24c1-48bc-a72e-54f34b18a318 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:40.236607 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e7bfc8-24c1-48bc-a72e-54f34b18a318 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:40.237160 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e7bfc8-24c1-48bc-a72e-54f34b18a318 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:40.237809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e7bfc8-24c1-48bc-a72e-54f34b18a318 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:40.239140 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e7bfc8-24c1-48bc-a72e-54f34b18a318 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:40.239839 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e7bfc8-24c1-48bc-a72e-54f34b18a318 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:40.242313 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e7bfc8-24c1-48bc-a72e-54f34b18a318 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:40.256026 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 112/223] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:40.263124 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e7bfc8-24c1-48bc-a72e-54f34b18a318 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:40.293442 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd81378-ab26-4614-ae20-3769d12bc8c5 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:40.341674 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d5efec-6259-4908-a74d-5eadcf38d981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:40.341674 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d5efec-6259-4908-a74d-5eadcf38d981 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:40.341674 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d5efec-6259-4908-a74d-5eadcf38d981 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:40.343549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d5efec-6259-4908-a74d-5eadcf38d981 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:40.413421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e7bfc8-24c1-48bc-a72e-54f34b18a318 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:40.451191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d5efec-6259-4908-a74d-5eadcf38d981 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:40.453230 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d5efec-6259-4908-a74d-5eadcf38d981 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:40.460433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e64f32-665d-4734-b6bc-db2f623140ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:40.461089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e64f32-665d-4734-b6bc-db2f623140ec None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:40.461417 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e64f32-665d-4734-b6bc-db2f623140ec None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:40.461673 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e64f32-665d-4734-b6bc-db2f623140ec None admin] PATH_INFO: `/v3/users/a74eaf6245b34eee909a0d40c27a95b8` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:40.462872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e64f32-665d-4734-b6bc-db2f623140ec None admin] RBAC: Authorizing `identity:get_user(user_id=a74eaf6245b34eee909a0d40c27a95b8)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:40.464927 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e64f32-665d-4734-b6bc-db2f623140ec None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:40.467966 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 113/224] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:40 2022] GET /identity/v3/users/a74eaf6245b34eee909a0d40c27a95b8 => generated 283 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:40.468941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e64f32-665d-4734-b6bc-db2f623140ec None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:40.485352 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:40.499604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:40.567894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 112/225] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:26:40.568852 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d5efec-6259-4908-a74d-5eadcf38d981 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:40.614068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e64f32-665d-4734-b6bc-db2f623140ec None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:40.667724 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea9b2e2-d76a-45a8-b8f5-7a62776b33ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:40.668290 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea9b2e2-d76a-45a8-b8f5-7a62776b33ef None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:40.668573 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea9b2e2-d76a-45a8-b8f5-7a62776b33ef None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:40.668850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea9b2e2-d76a-45a8-b8f5-7a62776b33ef None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:40.681903 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d5efec-6259-4908-a74d-5eadcf38d981 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:40.683223 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea9b2e2-d76a-45a8-b8f5-7a62776b33ef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:40.690635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-6ea9b2e2-d76a-45a8-b8f5-7a62776b33ef None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:26:40.690635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:40.690635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:40.690635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:40.690635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:40.690635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:40.690635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:40.690635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:40.690635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:40.690635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:40.690635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:40.690635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:40.690635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:40.690635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:40.690635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:40.690635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:40.691528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:40.691528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:40.691528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:40.691528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:40.691528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:40.691528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:40.691528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:40.691528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:40.691528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:40.691528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:40.691528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:40.691528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:40.691528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:40.691528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:40.691528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:40.691528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:40.692362 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:40.692362 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:40.692362 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:40.692362 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:40.692362 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:40.692362 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:40.692362 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:40.692362 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:40.692362 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:40.692362 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:40.692362 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:26:40.692362 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:40.692362 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 114/226] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:26:40 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:40.693654 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea9b2e2-d76a-45a8-b8f5-7a62776b33ef None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:40.711540 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:40.723761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902c2672-c10a-4cad-ad78-3ed1675e9510 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:40.724481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902c2672-c10a-4cad-ad78-3ed1675e9510 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:40.724764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902c2672-c10a-4cad-ad78-3ed1675e9510 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:40.725039 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902c2672-c10a-4cad-ad78-3ed1675e9510 None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:40.726266 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902c2672-c10a-4cad-ad78-3ed1675e9510 None admin] RBAC: Authorizing `identity:get_project(project_id=25655c9cb779428f96b5430e993f1337)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:40.727875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902c2672-c10a-4cad-ad78-3ed1675e9510 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:40.729693 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 113/227] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:40 2022] GET /identity/v3/projects/25655c9cb779428f96b5430e993f1337 => generated 302 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:40.730526 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902c2672-c10a-4cad-ad78-3ed1675e9510 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:40.793990 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:40.819095 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea9b2e2-d76a-45a8-b8f5-7a62776b33ef None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:40.855828 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669caf7a-a520-41a1-9f00-4a2841ab6f58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:40.856620 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669caf7a-a520-41a1-9f00-4a2841ab6f58 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:40.857002 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669caf7a-a520-41a1-9f00-4a2841ab6f58 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:40.857379 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669caf7a-a520-41a1-9f00-4a2841ab6f58 None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:40.858640 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669caf7a-a520-41a1-9f00-4a2841ab6f58 None admin] RBAC: Authorizing `identity:get_project(project_id=25655c9cb779428f96b5430e993f1337)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:40.858822 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902c2672-c10a-4cad-ad78-3ed1675e9510 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:40.860451 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669caf7a-a520-41a1-9f00-4a2841ab6f58 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:40.862465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 115/228] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:40 2022] GET /identity/v3/projects/25655c9cb779428f96b5430e993f1337 => generated 302 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:40.863183 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669caf7a-a520-41a1-9f00-4a2841ab6f58 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:40.899834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7918060f-2cb2-4f39-86ee-7159e010d088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:40.901478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7918060f-2cb2-4f39-86ee-7159e010d088 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:40.901948 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7918060f-2cb2-4f39-86ee-7159e010d088 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:40.902369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7918060f-2cb2-4f39-86ee-7159e010d088 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:40.903207 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7918060f-2cb2-4f39-86ee-7159e010d088 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:40.903658 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7918060f-2cb2-4f39-86ee-7159e010d088 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:40.905186 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7918060f-2cb2-4f39-86ee-7159e010d088 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:40.917869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 114/229] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:26:40 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:40.918946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7918060f-2cb2-4f39-86ee-7159e010d088 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:40.980277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669caf7a-a520-41a1-9f00-4a2841ab6f58 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:40.986497 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:41.034119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0156754-39ea-433c-b00c-2ddd218fad20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:41.035252 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0156754-39ea-433c-b00c-2ddd218fad20 None admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:41.036035 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0156754-39ea-433c-b00c-2ddd218fad20 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:41.036418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0156754-39ea-433c-b00c-2ddd218fad20 None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337/users/a74eaf6245b34eee909a0d40c27a95b8/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:41.038732 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0156754-39ea-433c-b00c-2ddd218fad20 None admin] RBAC: Authorizing `identity:create_grant(project_id=25655c9cb779428f96b5430e993f1337, user_id=a74eaf6245b34eee909a0d40c27a95b8, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:41.038937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7918060f-2cb2-4f39-86ee-7159e010d088 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:41.040196 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0156754-39ea-433c-b00c-2ddd218fad20 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:41.074718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c838dd-3f3c-4596-a1a2-dde80504e1ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:41.075484 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c838dd-3f3c-4596-a1a2-dde80504e1ef None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:41.075772 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c838dd-3f3c-4596-a1a2-dde80504e1ef None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:41.076045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c838dd-3f3c-4596-a1a2-dde80504e1ef None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:41.076763 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c838dd-3f3c-4596-a1a2-dde80504e1ef None admin] RBAC: Adding query filter params (role.id=025f58735650469d9fa6faca3e9e7998, scope.project.id=25655c9cb779428f96b5430e993f1337, user.id=01598eff6b08454eab252b4d1d3de8dd) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:41.077116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c838dd-3f3c-4596-a1a2-dde80504e1ef None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:41.079144 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c838dd-3f3c-4596-a1a2-dde80504e1ef None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:41.084822 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 115/230] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:26:40 2022] GET /identity/v3/role_assignments?user.id=01598eff6b08454eab252b4d1d3de8dd&scope.project.id=25655c9cb779428f96b5430e993f1337&role.id=025f58735650469d9fa6faca3e9e7998 => generated 259 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:41.085667 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c838dd-3f3c-4596-a1a2-dde80504e1ef None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:41.147232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 116/231] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:26:40 2022] PUT /identity/v3/projects/25655c9cb779428f96b5430e993f1337/users/a74eaf6245b34eee909a0d40c27a95b8/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:26:41.219675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c838dd-3f3c-4596-a1a2-dde80504e1ef None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:41.240598 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:41.260989 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92052a5e-db13-4c1e-baec-48bddf2b8a27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:41.261848 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92052a5e-db13-4c1e-baec-48bddf2b8a27 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:41.262131 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92052a5e-db13-4c1e-baec-48bddf2b8a27 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:41.262492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92052a5e-db13-4c1e-baec-48bddf2b8a27 None admin] PATH_INFO: `/v3/users/a74eaf6245b34eee909a0d40c27a95b8` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:41.263839 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92052a5e-db13-4c1e-baec-48bddf2b8a27 None admin] RBAC: Authorizing `identity:get_user(user_id=a74eaf6245b34eee909a0d40c27a95b8)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:41.265377 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92052a5e-db13-4c1e-baec-48bddf2b8a27 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:41.267635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 116/232] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:41 2022] GET /identity/v3/users/a74eaf6245b34eee909a0d40c27a95b8 => generated 283 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:41.274002 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0156754-39ea-433c-b00c-2ddd218fad20 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:41.283685 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:41.419141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0156754-39ea-433c-b00c-2ddd218fad20 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:41.463915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0909042-1049-4e58-9541-4541bd54cc64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:41.464489 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0909042-1049-4e58-9541-4541bd54cc64 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:41.464772 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0909042-1049-4e58-9541-4541bd54cc64 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:41.465049 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0909042-1049-4e58-9541-4541bd54cc64 None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:41.466199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0909042-1049-4e58-9541-4541bd54cc64 None admin] RBAC: Authorizing `identity:get_project(project_id=25655c9cb779428f96b5430e993f1337)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:41.467709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0909042-1049-4e58-9541-4541bd54cc64 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:41.469896 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 117/233] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:41 2022] GET /identity/v3/projects/25655c9cb779428f96b5430e993f1337 => generated 302 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:41.475764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92052a5e-db13-4c1e-baec-48bddf2b8a27 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:41.599366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92052a5e-db13-4c1e-baec-48bddf2b8a27 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:41.638083 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde7bef1-d84b-413b-a6a7-db7129c7eb4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:41.638780 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde7bef1-d84b-413b-a6a7-db7129c7eb4e None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:41.639063 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde7bef1-d84b-413b-a6a7-db7129c7eb4e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:41.639335 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde7bef1-d84b-413b-a6a7-db7129c7eb4e None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337/users/a74eaf6245b34eee909a0d40c27a95b8/roles/da6797f7fa24429095283b460be5c792` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:41.641594 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde7bef1-d84b-413b-a6a7-db7129c7eb4e None admin] RBAC: Authorizing `identity:create_grant(project_id=25655c9cb779428f96b5430e993f1337, user_id=a74eaf6245b34eee909a0d40c27a95b8, role_id=da6797f7fa24429095283b460be5c792)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:41.643077 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde7bef1-d84b-413b-a6a7-db7129c7eb4e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:41.754622 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 117/234] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:26:41 2022] PUT /identity/v3/projects/25655c9cb779428f96b5430e993f1337/users/a74eaf6245b34eee909a0d40c27a95b8/roles/da6797f7fa24429095283b460be5c792 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:26:42.690221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fff94df-3488-44f1-8662-ae8a34627315 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:42.690790 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fff94df-3488-44f1-8662-ae8a34627315 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:42.690790 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fff94df-3488-44f1-8662-ae8a34627315 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:42.692390 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 118/235] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:42.697779 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bb11ef-c6a0-4ec4-be9c-123f49710de6 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:42.698179 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bb11ef-c6a0-4ec4-be9c-123f49710de6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:42.698345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bb11ef-c6a0-4ec4-be9c-123f49710de6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:42.716394 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-18bb11ef-c6a0-4ec4-be9c-123f49710de6 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:42.717589 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-18bb11ef-c6a0-4ec4-be9c-123f49710de6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:42.772369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:42.808884 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 118/236] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:42.865231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9683c51b-5cf3-4ae6-a280-3b3b83bcceef None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:42.865865 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9683c51b-5cf3-4ae6-a280-3b3b83bcceef None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:42.866335 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9683c51b-5cf3-4ae6-a280-3b3b83bcceef None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:42.867915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 119/237] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:42.873285 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cb8533-3fa8-4b2e-b8f6-da9930922df4 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:42.873821 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cb8533-3fa8-4b2e-b8f6-da9930922df4 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:42.874289 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cb8533-3fa8-4b2e-b8f6-da9930922df4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:42.893996 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-27cb8533-3fa8-4b2e-b8f6-da9930922df4 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:42.895495 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-27cb8533-3fa8-4b2e-b8f6-da9930922df4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:42.971737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 119/238] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:43.255906 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9683c51b-5cf3-4ae6-a280-3b3b83bcceef None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:43.339315 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cb8533-3fa8-4b2e-b8f6-da9930922df4 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:43.355236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:43.415629 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9683c51b-5cf3-4ae6-a280-3b3b83bcceef None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:43.489935 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ea8f9a-efb7-4a43-a29c-f2d10df03a43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:43.490580 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ea8f9a-efb7-4a43-a29c-f2d10df03a43 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:43.490878 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ea8f9a-efb7-4a43-a29c-f2d10df03a43 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:43.491148 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ea8f9a-efb7-4a43-a29c-f2d10df03a43 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:43.513905 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ea8f9a-efb7-4a43-a29c-f2d10df03a43 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:43.519726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-c2ea8f9a-efb7-4a43-a29c-f2d10df03a43 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 30 09:26:43.519726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:43.519726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:43.519726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:43.519726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:43.519726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:43.519726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:43.519726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:43.519726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:43.519726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:43.519726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:43.519726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:43.519726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:43.519726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:43.519726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:43.519726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:43.520709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:43.520709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:43.520709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:43.520709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:43.520709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:43.520709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:43.520709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:43.520709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:43.520709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:43.520709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:43.520709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:43.520709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:43.520709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:43.520709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:43.520709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:43.520709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:43.521531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:43.521531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:43.521531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:43.521531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:43.521531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:43.521531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:43.521531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:43.521531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:43.521531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:43.521531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:43.521531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 30 09:26:43.521531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:43.521531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 120/239] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:43 2022] GET /identity/v3/roles/reader => generated 84 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:43.527498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ea8f9a-efb7-4a43-a29c-f2d10df03a43 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:43.551067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cb8533-3fa8-4b2e-b8f6-da9930922df4 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:43.554910 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:43.611782 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74521efa-0655-43fe-aed9-e2ac1fcc373d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:43.612907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74521efa-0655-43fe-aed9-e2ac1fcc373d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:43.613453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74521efa-0655-43fe-aed9-e2ac1fcc373d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:43.613978 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74521efa-0655-43fe-aed9-e2ac1fcc373d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:43.615320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74521efa-0655-43fe-aed9-e2ac1fcc373d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:43.618106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74521efa-0655-43fe-aed9-e2ac1fcc373d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:43.620514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 120/240] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:26:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:43.621847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e76f76c-e756-455e-b01a-e44a22df5ce3 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:43.622339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e76f76c-e756-455e-b01a-e44a22df5ce3 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:43.622849 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e76f76c-e756-455e-b01a-e44a22df5ce3 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:43.624294 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 121/241] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:43.629506 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e76f76c-e756-455e-b01a-e44a22df5ce3 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:43.667260 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ea8f9a-efb7-4a43-a29c-f2d10df03a43 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:43.699119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:43.705418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59812c9f-be93-4799-9a62-6f1fd6990613 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:43.706179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59812c9f-be93-4799-9a62-6f1fd6990613 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:43.706687 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59812c9f-be93-4799-9a62-6f1fd6990613 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:43.707063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59812c9f-be93-4799-9a62-6f1fd6990613 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:43.708405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59812c9f-be93-4799-9a62-6f1fd6990613 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:43.708405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59812c9f-be93-4799-9a62-6f1fd6990613 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:43.709999 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59812c9f-be93-4799-9a62-6f1fd6990613 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:43.722730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 121/242] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:43 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:43.723259 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103a654e-a8b0-49d2-8ee5-534f1c9c3999 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:43.723747 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103a654e-a8b0-49d2-8ee5-534f1c9c3999 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:43.724244 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103a654e-a8b0-49d2-8ee5-534f1c9c3999 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:43.745173 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-103a654e-a8b0-49d2-8ee5-534f1c9c3999 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:43.746885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-103a654e-a8b0-49d2-8ee5-534f1c9c3999 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:43.754559 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e76f76c-e756-455e-b01a-e44a22df5ce3 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:43.801689 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a2fe7f-f559-4290-b6bc-7f0eebcbcce0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:43.802909 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a2fe7f-f559-4290-b6bc-7f0eebcbcce0 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:43.803864 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a2fe7f-f559-4290-b6bc-7f0eebcbcce0 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:43.804366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a2fe7f-f559-4290-b6bc-7f0eebcbcce0 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:43.805402 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a2fe7f-f559-4290-b6bc-7f0eebcbcce0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:43.807787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a2fe7f-f559-4290-b6bc-7f0eebcbcce0 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:43.830666 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 122/243] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:43.830666 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103a654e-a8b0-49d2-8ee5-534f1c9c3999 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:43.920076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 122/244] 10.222.0.39 () {60 vars in 1132 bytes} [Fri Sep 30 09:26:43 2022] POST /identity/v3/users => generated 293 bytes in 291 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:26:43.928385 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7242f207-d3a2-4eea-b428-944a66261f36 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:43.928702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7242f207-d3a2-4eea-b428-944a66261f36 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:43.928941 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7242f207-d3a2-4eea-b428-944a66261f36 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:43.930353 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 123/245] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:43 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:43.936074 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103a654e-a8b0-49d2-8ee5-534f1c9c3999 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:43.936406 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a5c918-602d-4c8e-abd2-589c67da1a47 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:43.936695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a5c918-602d-4c8e-abd2-589c67da1a47 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:43.936972 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a5c918-602d-4c8e-abd2-589c67da1a47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:43.965954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:43.975839 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-68a5c918-602d-4c8e-abd2-589c67da1a47 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:43.984761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-68a5c918-602d-4c8e-abd2-589c67da1a47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:43.992387 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7246078-422a-4de1-8bf6-5b57bfcaa89d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:43.993122 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7246078-422a-4de1-8bf6-5b57bfcaa89d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:43.993402 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7246078-422a-4de1-8bf6-5b57bfcaa89d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:43.993672 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7246078-422a-4de1-8bf6-5b57bfcaa89d None admin] PATH_INFO: `/v3/users/24406c9ebcf448cbbe0517382694b044` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:44.022949 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7246078-422a-4de1-8bf6-5b57bfcaa89d None admin] RBAC: Authorizing `identity:get_user(user_id=24406c9ebcf448cbbe0517382694b044)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:44.024501 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7246078-422a-4de1-8bf6-5b57bfcaa89d None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:44.026625 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 123/246] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:43 2022] GET /identity/v3/users/24406c9ebcf448cbbe0517382694b044 => generated 297 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:44.034321 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:44.037294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7246078-422a-4de1-8bf6-5b57bfcaa89d None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:44.133649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 124/247] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:44.139472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a5c918-602d-4c8e-abd2-589c67da1a47 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:44.216405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7246078-422a-4de1-8bf6-5b57bfcaa89d None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:44.255089 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a5c918-602d-4c8e-abd2-589c67da1a47 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:44.262349 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35353e49-c312-4c94-8381-535704635acb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:44.262994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35353e49-c312-4c94-8381-535704635acb None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:44.263275 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35353e49-c312-4c94-8381-535704635acb None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:44.263549 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35353e49-c312-4c94-8381-535704635acb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:44.298151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5967cc1-8a08-4d01-9e5d-eea3186b6282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:44.298859 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5967cc1-8a08-4d01-9e5d-eea3186b6282 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:44.299144 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5967cc1-8a08-4d01-9e5d-eea3186b6282 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:44.299413 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5967cc1-8a08-4d01-9e5d-eea3186b6282 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:44.310898 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5967cc1-8a08-4d01-9e5d-eea3186b6282 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:44.318747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-a5967cc1-8a08-4d01-9e5d-eea3186b6282 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:44.318747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:44.318747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:44.318747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:44.318747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:44.318747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:44.318747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:44.318747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:44.318747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:44.318747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:44.318747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:44.318747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:44.318747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:44.318747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:44.318747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:44.318747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:44.319664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:44.319664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:44.319664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:44.319664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:44.319664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:44.319664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:44.319664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:44.319664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:44.319664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:44.319664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:44.319664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:44.319664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:44.319664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:44.319664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:44.319664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:44.319664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:44.320481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:44.320481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:44.320481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:44.320481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:44.320481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:44.320481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:44.320481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:44.320481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:44.320481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:44.320481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:44.320481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:44.320481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:26:44.320481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 125/248] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:44 2022] GET /identity/v3/roles/member => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:44.323113 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:44.327186 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5967cc1-8a08-4d01-9e5d-eea3186b6282 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:44.345543 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35353e49-c312-4c94-8381-535704635acb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:44.347200 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35353e49-c312-4c94-8381-535704635acb None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:44.352998 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:44.452324 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5967cc1-8a08-4d01-9e5d-eea3186b6282 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:44.485744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 124/249] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 449 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:26:44.490698 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35353e49-c312-4c94-8381-535704635acb None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:44.492463 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7d7a0d-ee9d-4cd2-9dc3-f578891d75ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:44.493189 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7d7a0d-ee9d-4cd2-9dc3-f578891d75ca None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:44.493472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7d7a0d-ee9d-4cd2-9dc3-f578891d75ca None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:44.493892 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7d7a0d-ee9d-4cd2-9dc3-f578891d75ca None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:44.494434 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7d7a0d-ee9d-4cd2-9dc3-f578891d75ca None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:44.494765 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7d7a0d-ee9d-4cd2-9dc3-f578891d75ca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:44.496215 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7d7a0d-ee9d-4cd2-9dc3-f578891d75ca None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:44.507161 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 126/250] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:44 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:44.508369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7d7a0d-ee9d-4cd2-9dc3-f578891d75ca None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:44.599516 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:44.606773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:44.621291 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35353e49-c312-4c94-8381-535704635acb None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:44.634262 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7d7a0d-ee9d-4cd2-9dc3-f578891d75ca None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:44.663571 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6551cec8-ae6f-4b43-a1cb-de29842b6e8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:44.663889 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6551cec8-ae6f-4b43-a1cb-de29842b6e8c None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:44.664124 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6551cec8-ae6f-4b43-a1cb-de29842b6e8c None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:44.664398 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6551cec8-ae6f-4b43-a1cb-de29842b6e8c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:44.676628 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700d9d2d-c733-4c71-8680-83224c8c0792 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:44.677274 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700d9d2d-c733-4c71-8680-83224c8c0792 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:44.677518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6551cec8-ae6f-4b43-a1cb-de29842b6e8c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:44.677622 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700d9d2d-c733-4c71-8680-83224c8c0792 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:44.677818 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700d9d2d-c733-4c71-8680-83224c8c0792 None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:44.679064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700d9d2d-c733-4c71-8680-83224c8c0792 None admin] RBAC: Authorizing `identity:get_project(project_id=25655c9cb779428f96b5430e993f1337)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:44.680585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700d9d2d-c733-4c71-8680-83224c8c0792 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:44.682307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 127/251] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:44 2022] GET /identity/v3/projects/25655c9cb779428f96b5430e993f1337 => generated 302 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:44.683181 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700d9d2d-c733-4c71-8680-83224c8c0792 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:44.684094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-6551cec8-ae6f-4b43-a1cb-de29842b6e8c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:26:44.684094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:44.684094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:44.684094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:44.684094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:44.684094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:44.684094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:44.684094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:44.684094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:44.684094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:44.684094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:44.684094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:44.684094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:44.684094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:44.684094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:44.684094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:44.685011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:44.685011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:44.685011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:44.685011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:44.685011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:44.685011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:44.685011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:44.685011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:44.685011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:44.685011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:44.685011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:44.685011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:44.685011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:44.685011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:44.685011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:44.685011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:44.685843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:44.685843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:44.685843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:44.685843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:44.685843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:44.685843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:44.685843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:44.685843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:44.685843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:44.685843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:44.685843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:26:44.685843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:44.685843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 125/252] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:26:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:44.686521 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0098b874-ae25-46b7-b20f-e5525d1f6dcf None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:44.687342 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0098b874-ae25-46b7-b20f-e5525d1f6dcf None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:44.687342 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0098b874-ae25-46b7-b20f-e5525d1f6dcf None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:44.688423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 126/253] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:44.696786 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f62b263-cede-4ab4-90d8-f503b6be028e None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:44.697072 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f62b263-cede-4ab4-90d8-f503b6be028e None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:44.697354 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f62b263-cede-4ab4-90d8-f503b6be028e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:44.721338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-0f62b263-cede-4ab4-90d8-f503b6be028e None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:44.723452 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0f62b263-cede-4ab4-90d8-f503b6be028e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:44.800447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700d9d2d-c733-4c71-8680-83224c8c0792 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:44.817144 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 127/254] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:44.818462 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f62b263-cede-4ab4-90d8-f503b6be028e None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:44.845263 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7157366-0fd1-465a-b703-09bd2b89eea8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:44.845906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7157366-0fd1-465a-b703-09bd2b89eea8 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:44.846182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7157366-0fd1-465a-b703-09bd2b89eea8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:44.846452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7157366-0fd1-465a-b703-09bd2b89eea8 None admin] PATH_INFO: `/v3/users/01598eff6b08454eab252b4d1d3de8dd` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:44.847604 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7157366-0fd1-465a-b703-09bd2b89eea8 None admin] RBAC: Authorizing `identity:get_user(user_id=01598eff6b08454eab252b4d1d3de8dd)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:44.849330 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7157366-0fd1-465a-b703-09bd2b89eea8 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:44.850927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 128/255] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:44 2022] GET /identity/v3/users/01598eff6b08454eab252b4d1d3de8dd => generated 297 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:44.851673 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7157366-0fd1-465a-b703-09bd2b89eea8 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:44.894948 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:44.965582 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:45.022301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f62b263-cede-4ab4-90d8-f503b6be028e None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:45.040262 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7157366-0fd1-465a-b703-09bd2b89eea8 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:45.081802 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceca2bb3-f44e-4700-a40d-bced4f9db81f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:45.082416 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceca2bb3-f44e-4700-a40d-bced4f9db81f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:45.082722 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceca2bb3-f44e-4700-a40d-bced4f9db81f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:45.084055 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceca2bb3-f44e-4700-a40d-bced4f9db81f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:45.084599 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceca2bb3-f44e-4700-a40d-bced4f9db81f None admin] RBAC: Adding query filter params (role.id=05a780eae85c41adab97f133b17a6ed4, scope.project.id=25655c9cb779428f96b5430e993f1337, user.id=24406c9ebcf448cbbe0517382694b044) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:45.084912 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceca2bb3-f44e-4700-a40d-bced4f9db81f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:45.086357 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceca2bb3-f44e-4700-a40d-bced4f9db81f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:45.092568 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9cfb78-22f2-4b3c-90ef-39cb522168b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:45.093182 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9cfb78-22f2-4b3c-90ef-39cb522168b1 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:45.093463 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9cfb78-22f2-4b3c-90ef-39cb522168b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:45.093736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9cfb78-22f2-4b3c-90ef-39cb522168b1 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:45.095004 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9cfb78-22f2-4b3c-90ef-39cb522168b1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:45.095309 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9cfb78-22f2-4b3c-90ef-39cb522168b1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:45.096684 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9cfb78-22f2-4b3c-90ef-39cb522168b1 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:45.102858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 129/256] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:26:44 2022] GET /identity/v3/role_assignments?user.id=24406c9ebcf448cbbe0517382694b044&scope.project.id=25655c9cb779428f96b5430e993f1337&role.id=05a780eae85c41adab97f133b17a6ed4 => generated 259 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:45.103701 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceca2bb3-f44e-4700-a40d-bced4f9db81f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:45.111179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 128/257] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:45.111939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9cfb78-22f2-4b3c-90ef-39cb522168b1 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:45.241245 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9cfb78-22f2-4b3c-90ef-39cb522168b1 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:45.278931 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:45.311943 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceca2bb3-f44e-4700-a40d-bced4f9db81f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:45.314787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad1ff8a-57b7-487c-9952-dae94872a96e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:45.315693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad1ff8a-57b7-487c-9952-dae94872a96e None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:45.316209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad1ff8a-57b7-487c-9952-dae94872a96e None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:45.316706 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad1ff8a-57b7-487c-9952-dae94872a96e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:45.326624 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:45.342651 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad1ff8a-57b7-487c-9952-dae94872a96e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:45.356008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-cad1ff8a-57b7-487c-9952-dae94872a96e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:26:45.356008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:45.356008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:45.356008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:45.356008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:45.356008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:45.356008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:45.356008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:45.356008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:45.356008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:45.356008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:45.356008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:45.356008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:45.356008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:45.356008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:45.356008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:45.357017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:45.357017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:45.357017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:45.357017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:45.357017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:45.357017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:45.357017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:45.357017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:45.357017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:45.357017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:45.357017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:45.357017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:45.357017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:45.357017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:45.357017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:45.357017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:45.357852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:45.357852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:45.357852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:45.357852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:45.357852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:45.357852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:45.357852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:45.357852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:45.357852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:45.357852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:45.357852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:26:45.357852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:45.357852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 129/258] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:26:45 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:45.358527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad1ff8a-57b7-487c-9952-dae94872a96e None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:45.366570 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b367a7-9152-4b2c-a19c-1ff6733e8343 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:45.367246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b367a7-9152-4b2c-a19c-1ff6733e8343 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:45.367523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b367a7-9152-4b2c-a19c-1ff6733e8343 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:45.367793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b367a7-9152-4b2c-a19c-1ff6733e8343 None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:45.369254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b367a7-9152-4b2c-a19c-1ff6733e8343 None admin] RBAC: Authorizing `identity:get_project(project_id=25655c9cb779428f96b5430e993f1337)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:45.370726 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b367a7-9152-4b2c-a19c-1ff6733e8343 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:45.372467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 130/259] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:45 2022] GET /identity/v3/projects/25655c9cb779428f96b5430e993f1337 => generated 302 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:45.379552 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b367a7-9152-4b2c-a19c-1ff6733e8343 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:45.485784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b367a7-9152-4b2c-a19c-1ff6733e8343 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:45.524727 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8db11f-60ca-4c9b-9c9f-7200cbafaa70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:45.525339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8db11f-60ca-4c9b-9c9f-7200cbafaa70 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:45.525614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8db11f-60ca-4c9b-9c9f-7200cbafaa70 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:45.525881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8db11f-60ca-4c9b-9c9f-7200cbafaa70 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:45.526533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8db11f-60ca-4c9b-9c9f-7200cbafaa70 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:45.526853 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8db11f-60ca-4c9b-9c9f-7200cbafaa70 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:45.528229 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8db11f-60ca-4c9b-9c9f-7200cbafaa70 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:45.534321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad1ff8a-57b7-487c-9952-dae94872a96e None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:45.547504 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 131/260] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:26:45 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:45.548137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8db11f-60ca-4c9b-9c9f-7200cbafaa70 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:45.579224 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:45.606467 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9521c686-0bd8-4892-8a1a-17f667d74432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:45.607144 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9521c686-0bd8-4892-8a1a-17f667d74432 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:45.607419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9521c686-0bd8-4892-8a1a-17f667d74432 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:45.607688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9521c686-0bd8-4892-8a1a-17f667d74432 None admin] PATH_INFO: `/v3/users/a74eaf6245b34eee909a0d40c27a95b8` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:45.610054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9521c686-0bd8-4892-8a1a-17f667d74432 None admin] RBAC: Authorizing `identity:get_user(user_id=a74eaf6245b34eee909a0d40c27a95b8)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:45.611552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9521c686-0bd8-4892-8a1a-17f667d74432 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:45.613123 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 130/261] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:45 2022] GET /identity/v3/users/a74eaf6245b34eee909a0d40c27a95b8 => generated 283 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:45.613870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9521c686-0bd8-4892-8a1a-17f667d74432 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:45.741334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:45.744366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8db11f-60ca-4c9b-9c9f-7200cbafaa70 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:45.805206 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c8b4d9-4455-475e-ba00-5cd828b8f733 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:45.806505 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c8b4d9-4455-475e-ba00-5cd828b8f733 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:45.806822 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c8b4d9-4455-475e-ba00-5cd828b8f733 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:45.807100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c8b4d9-4455-475e-ba00-5cd828b8f733 None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337/users/01598eff6b08454eab252b4d1d3de8dd/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:45.813952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c8b4d9-4455-475e-ba00-5cd828b8f733 None admin] RBAC: Authorizing `identity:create_grant(project_id=25655c9cb779428f96b5430e993f1337, user_id=01598eff6b08454eab252b4d1d3de8dd, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:45.815432 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c8b4d9-4455-475e-ba00-5cd828b8f733 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:45.821780 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9521c686-0bd8-4892-8a1a-17f667d74432 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:45.860910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b37d81-236a-42fd-b01a-06e95eb222e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:45.861840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b37d81-236a-42fd-b01a-06e95eb222e0 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:45.862131 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b37d81-236a-42fd-b01a-06e95eb222e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:45.862404 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b37d81-236a-42fd-b01a-06e95eb222e0 None admin] PATH_INFO: `/v3/users/a74eaf6245b34eee909a0d40c27a95b8` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:45.863574 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b37d81-236a-42fd-b01a-06e95eb222e0 None admin] RBAC: Authorizing `identity:get_user(user_id=a74eaf6245b34eee909a0d40c27a95b8)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:45.865026 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b37d81-236a-42fd-b01a-06e95eb222e0 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:45.866864 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 131/262] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:45 2022] GET /identity/v3/users/a74eaf6245b34eee909a0d40c27a95b8 => generated 283 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:45.867537 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b37d81-236a-42fd-b01a-06e95eb222e0 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:45.902696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 132/263] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:26:45 2022] PUT /identity/v3/projects/25655c9cb779428f96b5430e993f1337/users/01598eff6b08454eab252b4d1d3de8dd/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:26:45.903630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c8b4d9-4455-475e-ba00-5cd828b8f733 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:46.042968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:46.070223 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b37d81-236a-42fd-b01a-06e95eb222e0 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:46.097208 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c8b4d9-4455-475e-ba00-5cd828b8f733 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:46.132713 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:46.143385 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2434616a-4fa1-46e1-a1ef-2ee5a5876d29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:46.144019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2434616a-4fa1-46e1-a1ef-2ee5a5876d29 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:46.144293 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2434616a-4fa1-46e1-a1ef-2ee5a5876d29 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:46.144563 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2434616a-4fa1-46e1-a1ef-2ee5a5876d29 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:46.167443 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c9117e-e56d-4680-9d9b-43ed63a65b1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:46.168142 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c9117e-e56d-4680-9d9b-43ed63a65b1f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:46.168471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c9117e-e56d-4680-9d9b-43ed63a65b1f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:46.168745 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c9117e-e56d-4680-9d9b-43ed63a65b1f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:46.267492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2434616a-4fa1-46e1-a1ef-2ee5a5876d29 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:46.269123 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2434616a-4fa1-46e1-a1ef-2ee5a5876d29 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:46.327732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c9117e-e56d-4680-9d9b-43ed63a65b1f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:46.329669 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c9117e-e56d-4680-9d9b-43ed63a65b1f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:46.375779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 132/264] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 508 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:26:46.386539 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2434616a-4fa1-46e1-a1ef-2ee5a5876d29 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:46.429985 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:46.497221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:46.521355 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2434616a-4fa1-46e1-a1ef-2ee5a5876d29 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:46.530789 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 133/265] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 628 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:26:46.536994 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c9117e-e56d-4680-9d9b-43ed63a65b1f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:46.567226 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45b0465-b15d-429d-8ac2-dfbefde73711 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:46.567871 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45b0465-b15d-429d-8ac2-dfbefde73711 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:46.568153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45b0465-b15d-429d-8ac2-dfbefde73711 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:46.568429 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45b0465-b15d-429d-8ac2-dfbefde73711 None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:46.571133 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45b0465-b15d-429d-8ac2-dfbefde73711 None admin] RBAC: Authorizing `identity:get_project(project_id=25655c9cb779428f96b5430e993f1337)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:46.572771 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45b0465-b15d-429d-8ac2-dfbefde73711 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:46.574551 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 133/266] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:46 2022] GET /identity/v3/projects/25655c9cb779428f96b5430e993f1337 => generated 302 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:46.597465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45b0465-b15d-429d-8ac2-dfbefde73711 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:46.658082 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c9117e-e56d-4680-9d9b-43ed63a65b1f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:46.693298 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4284dd3e-14ba-49cb-aafa-15957bd6d716 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:46.694292 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4284dd3e-14ba-49cb-aafa-15957bd6d716 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:46.694852 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4284dd3e-14ba-49cb-aafa-15957bd6d716 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:46.695333 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4284dd3e-14ba-49cb-aafa-15957bd6d716 None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:46.696663 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4284dd3e-14ba-49cb-aafa-15957bd6d716 None admin] RBAC: Authorizing `identity:get_project(project_id=25655c9cb779428f96b5430e993f1337)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:46.698342 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4284dd3e-14ba-49cb-aafa-15957bd6d716 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:46.700806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 134/267] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:46 2022] GET /identity/v3/projects/25655c9cb779428f96b5430e993f1337 => generated 302 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:46.706798 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4284dd3e-14ba-49cb-aafa-15957bd6d716 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:46.718173 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:46.723088 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45b0465-b15d-429d-8ac2-dfbefde73711 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:46.765423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:46.774638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21382c36-60f7-4dca-9ba1-fa472cc8ccf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:46.775860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21382c36-60f7-4dca-9ba1-fa472cc8ccf3 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:46.776352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21382c36-60f7-4dca-9ba1-fa472cc8ccf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:46.779955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21382c36-60f7-4dca-9ba1-fa472cc8ccf3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:46.780890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21382c36-60f7-4dca-9ba1-fa472cc8ccf3 None admin] RBAC: Adding query filter params (role.id=e874c311084c4c9fb7691bce461d319b, scope.project.id=25655c9cb779428f96b5430e993f1337, user.id=a74eaf6245b34eee909a0d40c27a95b8) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:46.784848 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21382c36-60f7-4dca-9ba1-fa472cc8ccf3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:46.784848 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21382c36-60f7-4dca-9ba1-fa472cc8ccf3 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:46.801533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 134/268] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:26:46 2022] GET /identity/v3/role_assignments?user.id=a74eaf6245b34eee909a0d40c27a95b8&scope.project.id=25655c9cb779428f96b5430e993f1337&role.id=e874c311084c4c9fb7691bce461d319b => generated 608 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:46.867646 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4284dd3e-14ba-49cb-aafa-15957bd6d716 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:46.913448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc18e5d9-c018-4d05-a160-42eddacc84c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:46.914066 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc18e5d9-c018-4d05-a160-42eddacc84c1 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:46.914347 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc18e5d9-c018-4d05-a160-42eddacc84c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:46.914641 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc18e5d9-c018-4d05-a160-42eddacc84c1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:46.915373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc18e5d9-c018-4d05-a160-42eddacc84c1 None admin] RBAC: Adding query filter params (role.id=da6797f7fa24429095283b460be5c792, scope.project.id=25655c9cb779428f96b5430e993f1337, user.id=a74eaf6245b34eee909a0d40c27a95b8) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:46.915685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc18e5d9-c018-4d05-a160-42eddacc84c1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:46.917154 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc18e5d9-c018-4d05-a160-42eddacc84c1 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:46.928089 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 135/269] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:26:46 2022] GET /identity/v3/role_assignments?user.id=a74eaf6245b34eee909a0d40c27a95b8&scope.project.id=25655c9cb779428f96b5430e993f1337&role.id=da6797f7fa24429095283b460be5c792 => generated 608 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:47.096561 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da9f8da-adc0-4b95-bf71-17ca5d9f85e4 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:47.097203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da9f8da-adc0-4b95-bf71-17ca5d9f85e4 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:47.097675 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da9f8da-adc0-4b95-bf71-17ca5d9f85e4 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:47.099905 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 135/270] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:47.105310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460e8bd5-0957-4c40-ad55-07660700a117 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:47.105684 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460e8bd5-0957-4c40-ad55-07660700a117 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:47.105862 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460e8bd5-0957-4c40-ad55-07660700a117 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:47.129542 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-460e8bd5-0957-4c40-ad55-07660700a117 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:47.130691 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-460e8bd5-0957-4c40-ad55-07660700a117 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:47.211726 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 136/271] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:47.276666 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460a453f-d9b1-4201-b6a7-8938c0ef0441 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:47.276973 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460a453f-d9b1-4201-b6a7-8938c0ef0441 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:47.277227 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460a453f-d9b1-4201-b6a7-8938c0ef0441 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:47.278903 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 136/272] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:47.284523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b55f47-b713-40ff-98bf-aa27e2533af0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:47.284814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b55f47-b713-40ff-98bf-aa27e2533af0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:47.285095 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b55f47-b713-40ff-98bf-aa27e2533af0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:47.311549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-08b55f47-b713-40ff-98bf-aa27e2533af0 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:47.313318 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-08b55f47-b713-40ff-98bf-aa27e2533af0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:47.345981 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:47.397969 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 137/273] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:47.583803 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460a453f-d9b1-4201-b6a7-8938c0ef0441 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:47.679670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b55f47-b713-40ff-98bf-aa27e2533af0 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:47.718029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460a453f-d9b1-4201-b6a7-8938c0ef0441 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:47.768488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af42cd05-c22c-4423-9349-7051e890e55b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:47.769120 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af42cd05-c22c-4423-9349-7051e890e55b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:47.769396 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af42cd05-c22c-4423-9349-7051e890e55b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:47.769664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af42cd05-c22c-4423-9349-7051e890e55b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:47.783309 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b55f47-b713-40ff-98bf-aa27e2533af0 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:47.783492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af42cd05-c22c-4423-9349-7051e890e55b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:47.789594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-af42cd05-c22c-4423-9349-7051e890e55b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:47.789594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:47.789594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:47.789594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:47.789594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:47.789594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:47.789594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:47.789594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:47.789594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:47.789594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:47.789594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:47.789594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:47.789594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:47.789594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:47.789594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:47.789594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:47.790542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:47.790542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:47.790542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:47.790542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:47.790542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:47.790542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:47.790542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:47.790542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:47.790542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:47.790542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:47.790542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:47.790542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:47.790542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:47.790542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:47.790542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:47.790542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:47.791418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:47.791418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:47.791418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:47.791418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:47.791418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:47.791418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:47.791418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:47.791418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:47.791418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:47.791418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:47.791418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:47.791418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:47.791418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 137/274] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:47.798790 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af42cd05-c22c-4423-9349-7051e890e55b None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:47.819804 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e141536c-5312-4915-9c51-cdd96ed98ab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:47.820441 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e141536c-5312-4915-9c51-cdd96ed98ab5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:47.820727 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e141536c-5312-4915-9c51-cdd96ed98ab5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:47.820999 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e141536c-5312-4915-9c51-cdd96ed98ab5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:47.822255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e141536c-5312-4915-9c51-cdd96ed98ab5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:47.824039 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e141536c-5312-4915-9c51-cdd96ed98ab5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:47.826359 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 138/275] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:26:47 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:47.832306 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e141536c-5312-4915-9c51-cdd96ed98ab5 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:47.833341 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:47.867348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:47.912624 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af42cd05-c22c-4423-9349-7051e890e55b None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:47.944212 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e141536c-5312-4915-9c51-cdd96ed98ab5 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:47.950944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2439d84-6f6e-4329-8a7f-5ae09ccab8db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:47.951670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2439d84-6f6e-4329-8a7f-5ae09ccab8db None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:47.952078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2439d84-6f6e-4329-8a7f-5ae09ccab8db None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:47.952418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2439d84-6f6e-4329-8a7f-5ae09ccab8db None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:47.953273 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2439d84-6f6e-4329-8a7f-5ae09ccab8db None admin] RBAC: Adding query filter params (name=member) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:47.953643 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2439d84-6f6e-4329-8a7f-5ae09ccab8db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:47.955729 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2439d84-6f6e-4329-8a7f-5ae09ccab8db None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:47.965973 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 138/276] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:47.976347 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2439d84-6f6e-4329-8a7f-5ae09ccab8db None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:47.986866 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a51ce5-7180-4fa8-8ea5-b859567b4092 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:47.987692 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a51ce5-7180-4fa8-8ea5-b859567b4092 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:47.988111 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a51ce5-7180-4fa8-8ea5-b859567b4092 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:47.988544 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a51ce5-7180-4fa8-8ea5-b859567b4092 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:47.989726 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a51ce5-7180-4fa8-8ea5-b859567b4092 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:47.991685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a51ce5-7180-4fa8-8ea5-b859567b4092 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:48.074666 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:48.089445 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 139/277] 10.222.0.39 () {60 vars in 1132 bytes} [Fri Sep 30 09:26:47 2022] POST /identity/v3/users => generated 293 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:26:48.090628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2439d84-6f6e-4329-8a7f-5ae09ccab8db None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:48.090768 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699029ca-5a6c-4244-b088-0dc328bc5596 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:48.094640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699029ca-5a6c-4244-b088-0dc328bc5596 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:48.094640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699029ca-5a6c-4244-b088-0dc328bc5596 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:48.094640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 140/278] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:48.100180 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955db62e-28c1-4259-bf02-6819735490b3 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:48.100494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955db62e-28c1-4259-bf02-6819735490b3 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:48.100800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955db62e-28c1-4259-bf02-6819735490b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:48.120994 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-955db62e-28c1-4259-bf02-6819735490b3 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:48.122312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-955db62e-28c1-4259-bf02-6819735490b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:48.135489 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761cba1d-1d9d-4eca-aa8b-3b28145fe01b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:48.136539 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761cba1d-1d9d-4eca-aa8b-3b28145fe01b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:48.137002 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761cba1d-1d9d-4eca-aa8b-3b28145fe01b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:48.137418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761cba1d-1d9d-4eca-aa8b-3b28145fe01b None admin] PATH_INFO: `/v3/users/41eca216f4d3472cae8a134cd4ee80ed` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:48.153951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761cba1d-1d9d-4eca-aa8b-3b28145fe01b None admin] RBAC: Authorizing `identity:get_user(user_id=41eca216f4d3472cae8a134cd4ee80ed)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:48.155736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761cba1d-1d9d-4eca-aa8b-3b28145fe01b None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:48.157675 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 139/279] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:47 2022] GET /identity/v3/users/41eca216f4d3472cae8a134cd4ee80ed => generated 293 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:48.163994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761cba1d-1d9d-4eca-aa8b-3b28145fe01b None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:48.216942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 141/280] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:26:48.287405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761cba1d-1d9d-4eca-aa8b-3b28145fe01b None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:48.324758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e937f5-2c42-426c-9006-57afe50d88a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:48.325327 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e937f5-2c42-426c-9006-57afe50d88a6 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:48.325611 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e937f5-2c42-426c-9006-57afe50d88a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:48.325884 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e937f5-2c42-426c-9006-57afe50d88a6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:48.326646 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e937f5-2c42-426c-9006-57afe50d88a6 None admin] RBAC: Adding query filter params (role.id=025f58735650469d9fa6faca3e9e7998, user.id=41eca216f4d3472cae8a134cd4ee80ed, scope.system=all) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:48.326959 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e937f5-2c42-426c-9006-57afe50d88a6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:48.328467 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e937f5-2c42-426c-9006-57afe50d88a6 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:48.333922 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 140/281] 10.222.0.39 () {56 vars in 1308 bytes} [Fri Sep 30 09:26:48 2022] GET /identity/v3/role_assignments?user.id=41eca216f4d3472cae8a134cd4ee80ed&scope.system=all&role.id=025f58735650469d9fa6faca3e9e7998 => generated 226 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:48.491986 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955db62e-28c1-4259-bf02-6819735490b3 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:48.512097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:48.602654 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955db62e-28c1-4259-bf02-6819735490b3 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:48.650480 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5b0b05-0278-4931-8cbe-52959b5dd796 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:48.651539 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5b0b05-0278-4931-8cbe-52959b5dd796 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:48.652012 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5b0b05-0278-4931-8cbe-52959b5dd796 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:48.652476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5b0b05-0278-4931-8cbe-52959b5dd796 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:48.671477 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5b0b05-0278-4931-8cbe-52959b5dd796 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:48.676879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-af5b0b05-0278-4931-8cbe-52959b5dd796 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 30 09:26:48.676879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:48.676879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:48.676879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:48.676879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:48.676879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:48.676879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:48.676879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:48.676879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:48.676879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:48.676879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:48.676879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:48.676879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:48.676879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:48.676879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:48.676879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:48.677749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:48.677749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:48.677749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:48.677749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:48.677749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:48.677749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:48.677749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:48.677749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:48.677749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:48.677749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:48.677749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:48.677749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:48.677749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:48.677749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:48.677749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:48.677749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:48.678549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:48.678549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:48.678549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:48.678549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:48.678549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:48.678549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:48.678549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:48.678549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:48.678549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:48.678549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:48.678549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 30 09:26:48.678549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:26:48.680812 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 142/282] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:48 2022] GET /identity/v3/roles/reader => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:48.686715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e937f5-2c42-426c-9006-57afe50d88a6 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:48.706266 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:48.776357 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df115ff-aa66-490c-a74c-9baf8e2d5954 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:48.776974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df115ff-aa66-490c-a74c-9baf8e2d5954 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:48.777443 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df115ff-aa66-490c-a74c-9baf8e2d5954 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:48.779709 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 143/283] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:48.789915 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3993e5-27ff-45f2-892b-779bd148ca62 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:48.790195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3993e5-27ff-45f2-892b-779bd148ca62 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:48.791050 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3993e5-27ff-45f2-892b-779bd148ca62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:48.818900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-8f3993e5-27ff-45f2-892b-779bd148ca62 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:48.820101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8f3993e5-27ff-45f2-892b-779bd148ca62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:48.861880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e937f5-2c42-426c-9006-57afe50d88a6 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:48.910657 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 144/284] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:48.924411 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c788c161-171b-4b07-ab30-d30f4dba1c75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:48.925026 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c788c161-171b-4b07-ab30-d30f4dba1c75 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:48.925312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c788c161-171b-4b07-ab30-d30f4dba1c75 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:48.925589 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c788c161-171b-4b07-ab30-d30f4dba1c75 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:48.926261 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c788c161-171b-4b07-ab30-d30f4dba1c75 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:48.926568 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c788c161-171b-4b07-ab30-d30f4dba1c75 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:48.927994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c788c161-171b-4b07-ab30-d30f4dba1c75 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:48.942630 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 141/285] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:48 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:48.948283 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3993e5-27ff-45f2-892b-779bd148ca62 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:48.958138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:49.081412 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3993e5-27ff-45f2-892b-779bd148ca62 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:49.124341 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79376ed-d358-443a-81b9-b851a1340966 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:49.125506 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79376ed-d358-443a-81b9-b851a1340966 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:49.125791 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79376ed-d358-443a-81b9-b851a1340966 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:49.126067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79376ed-d358-443a-81b9-b851a1340966 None admin] PATH_INFO: `/v3/users/24406c9ebcf448cbbe0517382694b044` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:49.127226 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79376ed-d358-443a-81b9-b851a1340966 None admin] RBAC: Authorizing `identity:get_user(user_id=24406c9ebcf448cbbe0517382694b044)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:49.128918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79376ed-d358-443a-81b9-b851a1340966 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:49.130892 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 145/286] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:48 2022] GET /identity/v3/users/24406c9ebcf448cbbe0517382694b044 => generated 297 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:49.139682 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c788c161-171b-4b07-ab30-d30f4dba1c75 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:49.239884 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:49.280229 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c788c161-171b-4b07-ab30-d30f4dba1c75 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:49.315588 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79376ed-d358-443a-81b9-b851a1340966 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:49.339239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56b27a6-b61c-40bc-9b00-acc18f63a18f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:49.339782 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56b27a6-b61c-40bc-9b00-acc18f63a18f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:49.340055 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56b27a6-b61c-40bc-9b00-acc18f63a18f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:49.340324 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56b27a6-b61c-40bc-9b00-acc18f63a18f None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:49.342151 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56b27a6-b61c-40bc-9b00-acc18f63a18f None admin] RBAC: Authorizing `identity:get_project(project_id=25655c9cb779428f96b5430e993f1337)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:49.343638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56b27a6-b61c-40bc-9b00-acc18f63a18f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:49.351001 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 142/287] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:49 2022] GET /identity/v3/projects/25655c9cb779428f96b5430e993f1337 => generated 302 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:49.354107 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56b27a6-b61c-40bc-9b00-acc18f63a18f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:49.459494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:49.512749 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56b27a6-b61c-40bc-9b00-acc18f63a18f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:49.517986 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79376ed-d358-443a-81b9-b851a1340966 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:49.553261 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afe6d32-f944-4e73-bdc3-258c6e2170fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:49.554011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afe6d32-f944-4e73-bdc3-258c6e2170fe None admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:49.554290 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afe6d32-f944-4e73-bdc3-258c6e2170fe None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:49.554392 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699b2fa5-ca58-409d-ae86-af270cbe773b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:49.554567 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afe6d32-f944-4e73-bdc3-258c6e2170fe None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337/users/24406c9ebcf448cbbe0517382694b044/roles/05a780eae85c41adab97f133b17a6ed4` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:49.555053 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699b2fa5-ca58-409d-ae86-af270cbe773b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:49.555334 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699b2fa5-ca58-409d-ae86-af270cbe773b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:49.555603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699b2fa5-ca58-409d-ae86-af270cbe773b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:49.556916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afe6d32-f944-4e73-bdc3-258c6e2170fe None admin] RBAC: Authorizing `identity:create_grant(project_id=25655c9cb779428f96b5430e993f1337, user_id=24406c9ebcf448cbbe0517382694b044, role_id=05a780eae85c41adab97f133b17a6ed4)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:49.559216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afe6d32-f944-4e73-bdc3-258c6e2170fe None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:49.567008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:49.570422 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699b2fa5-ca58-409d-ae86-af270cbe773b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:49.575927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-699b2fa5-ca58-409d-ae86-af270cbe773b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:49.575927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:49.575927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:49.575927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:49.575927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:49.575927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:49.575927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:49.575927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:49.575927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:49.575927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:49.575927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:49.575927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:49.575927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:49.575927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:49.575927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:49.575927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:49.576792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:49.576792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:49.576792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:49.576792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:49.576792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:49.576792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:49.576792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:49.576792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:49.576792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:49.576792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:49.576792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:49.576792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:49.576792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:49.576792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:49.576792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:49.576792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:49.577846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:49.577846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:49.577846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:49.577846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:49.577846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:49.577846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:49.577846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:49.577846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:49.577846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:49.577846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:49.577846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:49.577846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:26:49.577846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 146/288] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:26:49.585272 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699b2fa5-ca58-409d-ae86-af270cbe773b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:49.651402 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 143/289] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:26:49 2022] PUT /identity/v3/projects/25655c9cb779428f96b5430e993f1337/users/24406c9ebcf448cbbe0517382694b044/roles/05a780eae85c41adab97f133b17a6ed4 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:26:49.697405 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699b2fa5-ca58-409d-ae86-af270cbe773b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:49.734139 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b4e09d-8cea-45e2-ada0-a470634293ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:49.735331 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b4e09d-8cea-45e2-ada0-a470634293ee None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:49.735607 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b4e09d-8cea-45e2-ada0-a470634293ee None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:49.735875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b4e09d-8cea-45e2-ada0-a470634293ee None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:49.736527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b4e09d-8cea-45e2-ada0-a470634293ee None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:49.736830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b4e09d-8cea-45e2-ada0-a470634293ee None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:49.738262 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b4e09d-8cea-45e2-ada0-a470634293ee None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:49.739723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:49.748704 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 147/290] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:49 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:49.756069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afe6d32-f944-4e73-bdc3-258c6e2170fe None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:49.886099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afe6d32-f944-4e73-bdc3-258c6e2170fe None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:49.930999 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0422e301-f858-41af-b8e3-6e62ab775ebe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:49.931580 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0422e301-f858-41af-b8e3-6e62ab775ebe None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:49.931847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0422e301-f858-41af-b8e3-6e62ab775ebe None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:49.932117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0422e301-f858-41af-b8e3-6e62ab775ebe None admin] PATH_INFO: `/v3/users/01598eff6b08454eab252b4d1d3de8dd` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:49.933358 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0422e301-f858-41af-b8e3-6e62ab775ebe None admin] RBAC: Authorizing `identity:get_user(user_id=01598eff6b08454eab252b4d1d3de8dd)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:49.934875 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0422e301-f858-41af-b8e3-6e62ab775ebe None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:49.936803 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 144/291] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:49 2022] GET /identity/v3/users/01598eff6b08454eab252b4d1d3de8dd => generated 297 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:49.942191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b4e09d-8cea-45e2-ada0-a470634293ee None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:50.065832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b4e09d-8cea-45e2-ada0-a470634293ee None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:50.142157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfd9b14-ae96-4150-92f3-6d85c98a524f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:50.142802 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfd9b14-ae96-4150-92f3-6d85c98a524f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:50.143086 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfd9b14-ae96-4150-92f3-6d85c98a524f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:50.143362 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfd9b14-ae96-4150-92f3-6d85c98a524f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:50.229283 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:50.240366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfd9b14-ae96-4150-92f3-6d85c98a524f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:50.242133 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfd9b14-ae96-4150-92f3-6d85c98a524f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:50.386247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 148/292] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:26:50.392111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0422e301-f858-41af-b8e3-6e62ab775ebe None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:50.471640 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:50.519455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0422e301-f858-41af-b8e3-6e62ab775ebe None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:50.575317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5270abe-246e-4362-b4b6-9f2d5e44231d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:50.575958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5270abe-246e-4362-b4b6-9f2d5e44231d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:50.576238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5270abe-246e-4362-b4b6-9f2d5e44231d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:50.576511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5270abe-246e-4362-b4b6-9f2d5e44231d None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:50.577666 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5270abe-246e-4362-b4b6-9f2d5e44231d None admin] RBAC: Authorizing `identity:get_project(project_id=25655c9cb779428f96b5430e993f1337)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:50.579706 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5270abe-246e-4362-b4b6-9f2d5e44231d None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:50.581693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 145/293] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:50 2022] GET /identity/v3/projects/25655c9cb779428f96b5430e993f1337 => generated 302 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:50.587259 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfd9b14-ae96-4150-92f3-6d85c98a524f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:50.704934 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:50.712509 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfd9b14-ae96-4150-92f3-6d85c98a524f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:50.754210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abef4409-04ee-4838-a772-58bcaf1d7eca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:50.754772 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abef4409-04ee-4838-a772-58bcaf1d7eca None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:50.755056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abef4409-04ee-4838-a772-58bcaf1d7eca None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:50.755332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abef4409-04ee-4838-a772-58bcaf1d7eca None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:50.756058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abef4409-04ee-4838-a772-58bcaf1d7eca None admin] RBAC: Adding query filter params (role.id=025f58735650469d9fa6faca3e9e7998, scope.project.id=25655c9cb779428f96b5430e993f1337, user.id=01598eff6b08454eab252b4d1d3de8dd) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:50.756372 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abef4409-04ee-4838-a772-58bcaf1d7eca None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:50.758102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abef4409-04ee-4838-a772-58bcaf1d7eca None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:50.769919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 149/294] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:26:50 2022] GET /identity/v3/role_assignments?user.id=01598eff6b08454eab252b4d1d3de8dd&scope.project.id=25655c9cb779428f96b5430e993f1337&role.id=025f58735650469d9fa6faca3e9e7998 => generated 608 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:50.931335 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcda377-1ea6-47d6-abb6-f5c588c00077 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:50.931846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcda377-1ea6-47d6-abb6-f5c588c00077 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:50.931846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcda377-1ea6-47d6-abb6-f5c588c00077 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:50.933469 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 146/295] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:50.939619 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f7eece-890a-475e-809f-a3c788440c5b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:50.939909 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f7eece-890a-475e-809f-a3c788440c5b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:50.940355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f7eece-890a-475e-809f-a3c788440c5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:50.967793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c0f7eece-890a-475e-809f-a3c788440c5b None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:50.969320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c0f7eece-890a-475e-809f-a3c788440c5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:51.010764 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56dd2f9-7f4c-404b-915b-37b844731da6 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:51.010764 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56dd2f9-7f4c-404b-915b-37b844731da6 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:51.010764 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56dd2f9-7f4c-404b-915b-37b844731da6 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:51.011784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 147/296] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:51.026862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1e1b32-0fdd-4c01-b1d0-4a51d8737b04 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:51.027438 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1e1b32-0fdd-4c01-b1d0-4a51d8737b04 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:51.027904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1e1b32-0fdd-4c01-b1d0-4a51d8737b04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:51.050772 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 150/297] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:51.059297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-4e1e1b32-0fdd-4c01-b1d0-4a51d8737b04 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:51.060831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-4e1e1b32-0fdd-4c01-b1d0-4a51d8737b04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:51.087550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f30fe0-70d6-46c6-a881-71ad8d441409 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:51.088505 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f30fe0-70d6-46c6-a881-71ad8d441409 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:51.089017 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f30fe0-70d6-46c6-a881-71ad8d441409 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:51.090893 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 151/298] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:26:51.096000 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e3c23f-77c7-458c-872e-5790023a7d22 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:51.096527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e3c23f-77c7-458c-872e-5790023a7d22 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:51.097003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e3c23f-77c7-458c-872e-5790023a7d22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:51.120997 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f3e3c23f-77c7-458c-872e-5790023a7d22 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:51.122501 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f3e3c23f-77c7-458c-872e-5790023a7d22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:51.153825 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 148/299] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:26:51.203060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 152/300] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:51.354956 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1e1b32-0fdd-4c01-b1d0-4a51d8737b04 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:51.429194 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e3c23f-77c7-458c-872e-5790023a7d22 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:51.432426 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:51.455074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:51.489283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1e1b32-0fdd-4c01-b1d0-4a51d8737b04 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:51.524622 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04202fd-6c79-4038-9ae8-e642131214b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:51.525378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04202fd-6c79-4038-9ae8-e642131214b4 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:51.525659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04202fd-6c79-4038-9ae8-e642131214b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:51.525929 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04202fd-6c79-4038-9ae8-e642131214b4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:51.537735 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04202fd-6c79-4038-9ae8-e642131214b4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:51.543070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-b04202fd-6c79-4038-9ae8-e642131214b4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:51.543070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:51.543070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:51.543070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:51.543070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:51.543070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:51.543070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:51.543070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:51.543070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:51.543070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:51.543070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:51.543070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:51.543070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:51.543070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:51.543070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:51.543070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:51.543976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:51.543976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:51.543976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:51.543976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:51.543976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:51.543976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:51.543976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:51.543976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:51.543976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:51.543976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:51.543976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:51.543976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:51.543976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:51.543976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:51.543976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:51.543976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:51.544817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:51.544817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:51.544817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:51.544817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:51.544817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:51.544817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:51.544817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:51.544817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:51.544817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:51.544817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:51.544817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:51.544817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:51.545433 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e3c23f-77c7-458c-872e-5790023a7d22 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:51.549533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 149/301] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:51.550265 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04202fd-6c79-4038-9ae8-e642131214b4 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:51.578759 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c5a78e-577b-4571-abe4-b07614ff89bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:51.579385 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c5a78e-577b-4571-abe4-b07614ff89bc None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:51.579662 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c5a78e-577b-4571-abe4-b07614ff89bc None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:51.579934 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c5a78e-577b-4571-abe4-b07614ff89bc None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:51.589631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c5a78e-577b-4571-abe4-b07614ff89bc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:51.595065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-b6c5a78e-577b-4571-abe4-b07614ff89bc None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 30 09:26:51.595065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:51.595065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:51.595065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:51.595065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:51.595065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:51.595065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:51.595065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:51.595065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:51.595065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:51.595065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:51.595065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:51.595065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:51.595065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:51.595065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:51.595065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:51.596003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:51.596003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:51.596003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:51.596003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:51.596003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:51.596003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:51.596003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:51.596003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:51.596003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:51.596003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:51.596003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:51.596003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:51.596003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:51.596003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:51.596003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:51.596003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:51.596834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:51.596834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:51.596834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:51.596834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:51.596834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:51.596834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:51.596834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:51.596834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:51.596834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:51.596834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:51.596834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 30 09:26:51.596834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:26:51.596834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 153/302] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:51 2022] GET /identity/v3/roles/reader => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:51.598031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c5a78e-577b-4571-abe4-b07614ff89bc None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:51.661140 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04202fd-6c79-4038-9ae8-e642131214b4 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:51.682342 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:51.684897 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:51.695795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b71873a-3c78-4735-ba99-8b1ac1734b0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:51.696422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b71873a-3c78-4735-ba99-8b1ac1734b0d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:51.696702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b71873a-3c78-4735-ba99-8b1ac1734b0d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:51.696973 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b71873a-3c78-4735-ba99-8b1ac1734b0d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:51.698068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b71873a-3c78-4735-ba99-8b1ac1734b0d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:51.699544 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b71873a-3c78-4735-ba99-8b1ac1734b0d None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:51.700877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c5a78e-577b-4571-abe4-b07614ff89bc None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:51.701093 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 150/303] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:26:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:51.703318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b71873a-3c78-4735-ba99-8b1ac1734b0d None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:51.737582 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d26549a-1353-4bd7-b94a-f33c004d9aa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:51.738195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d26549a-1353-4bd7-b94a-f33c004d9aa2 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:51.738478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d26549a-1353-4bd7-b94a-f33c004d9aa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:51.738787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d26549a-1353-4bd7-b94a-f33c004d9aa2 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:51.739445 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d26549a-1353-4bd7-b94a-f33c004d9aa2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:51.739750 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d26549a-1353-4bd7-b94a-f33c004d9aa2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:51.741180 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d26549a-1353-4bd7-b94a-f33c004d9aa2 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:51.749447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 154/304] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:51.751458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d26549a-1353-4bd7-b94a-f33c004d9aa2 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:51.813265 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b71873a-3c78-4735-ba99-8b1ac1734b0d None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:51.850367 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9213cc71-a17b-4974-a925-16464caa0c06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:51.851005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9213cc71-a17b-4974-a925-16464caa0c06 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:51.851290 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9213cc71-a17b-4974-a925-16464caa0c06 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:51.851567 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9213cc71-a17b-4974-a925-16464caa0c06 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:51.852228 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9213cc71-a17b-4974-a925-16464caa0c06 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:51.852531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9213cc71-a17b-4974-a925-16464caa0c06 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:51.853917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9213cc71-a17b-4974-a925-16464caa0c06 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:51.860671 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d26549a-1353-4bd7-b94a-f33c004d9aa2 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:51.867363 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 151/305] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:51 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:51.867363 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9213cc71-a17b-4974-a925-16464caa0c06 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:51.901694 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcbb884-f490-40dc-95aa-d0429adedec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:51.902312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcbb884-f490-40dc-95aa-d0429adedec8 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:51.902617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcbb884-f490-40dc-95aa-d0429adedec8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:51.902889 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcbb884-f490-40dc-95aa-d0429adedec8 None admin] PATH_INFO: `/v3/groups` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:51.903509 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edcbb884-f490-40dc-95aa-d0429adedec8 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:51.904941 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edcbb884-f490-40dc-95aa-d0429adedec8 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:51.919338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:51.966642 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9213cc71-a17b-4974-a925-16464caa0c06 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:51.977009 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 155/306] 10.222.0.39 () {60 vars in 1134 bytes} [Fri Sep 30 09:26:51 2022] POST /identity/v3/groups => generated 229 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:26:51.978002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcbb884-f490-40dc-95aa-d0429adedec8 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:51.980846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:52.019045 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790cec9e-6008-43c8-b2ff-048274abdf2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:52.020191 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790cec9e-6008-43c8-b2ff-048274abdf2d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:52.020599 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790cec9e-6008-43c8-b2ff-048274abdf2d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:52.020929 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790cec9e-6008-43c8-b2ff-048274abdf2d None admin] PATH_INFO: `/v3/users/41eca216f4d3472cae8a134cd4ee80ed` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:52.022552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790cec9e-6008-43c8-b2ff-048274abdf2d None admin] RBAC: Authorizing `identity:get_user(user_id=41eca216f4d3472cae8a134cd4ee80ed)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:52.025112 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790cec9e-6008-43c8-b2ff-048274abdf2d None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:52.033063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 152/307] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:51 2022] GET /identity/v3/users/41eca216f4d3472cae8a134cd4ee80ed => generated 293 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:52.035621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790cec9e-6008-43c8-b2ff-048274abdf2d None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:52.146997 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcbb884-f490-40dc-95aa-d0429adedec8 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:52.164020 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790cec9e-6008-43c8-b2ff-048274abdf2d None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:52.191141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:52.196110 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936bcb76-954e-4d35-bc43-6940ba472e71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:52.196762 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936bcb76-954e-4d35-bc43-6940ba472e71 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:52.197037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936bcb76-954e-4d35-bc43-6940ba472e71 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:52.197306 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936bcb76-954e-4d35-bc43-6940ba472e71 None admin] PATH_INFO: `/v3/users/9435e491ae594deab29b230ea7157a34` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:52.215310 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fef87c-b0ae-48c4-8b3c-8156e7c19df6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:52.215703 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936bcb76-954e-4d35-bc43-6940ba472e71 None admin] RBAC: Authorizing `identity:get_user(user_id=9435e491ae594deab29b230ea7157a34)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:52.215970 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fef87c-b0ae-48c4-8b3c-8156e7c19df6 None admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:52.216248 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fef87c-b0ae-48c4-8b3c-8156e7c19df6 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:52.216520 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fef87c-b0ae-48c4-8b3c-8156e7c19df6 None admin] PATH_INFO: `/v3/system/users/41eca216f4d3472cae8a134cd4ee80ed/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:52.217328 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936bcb76-954e-4d35-bc43-6940ba472e71 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:52.217921 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8fef87c-b0ae-48c4-8b3c-8156e7c19df6 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=41eca216f4d3472cae8a134cd4ee80ed, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:52.219358 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8fef87c-b0ae-48c4-8b3c-8156e7c19df6 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:52.219463 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 156/308] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:51 2022] GET /identity/v3/users/9435e491ae594deab29b230ea7157a34 => generated 293 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:52.226872 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936bcb76-954e-4d35-bc43-6940ba472e71 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:52.281206 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 153/309] 10.222.0.39 () {58 vars in 1334 bytes} [Fri Sep 30 09:26:52 2022] PUT /identity/v3/system/users/41eca216f4d3472cae8a134cd4ee80ed/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:26:52.302010 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:52.354190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936bcb76-954e-4d35-bc43-6940ba472e71 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:52.395285 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c8e561-b792-4739-a408-0c8d6948e256 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:52.395912 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c8e561-b792-4739-a408-0c8d6948e256 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:52.396192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c8e561-b792-4739-a408-0c8d6948e256 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:52.396467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c8e561-b792-4739-a408-0c8d6948e256 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:52.397314 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c8e561-b792-4739-a408-0c8d6948e256 None admin] RBAC: Adding query filter params (role.id=05a780eae85c41adab97f133b17a6ed4, user.id=9435e491ae594deab29b230ea7157a34, scope.system=all) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:52.397632 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c8e561-b792-4739-a408-0c8d6948e256 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:52.399172 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c8e561-b792-4739-a408-0c8d6948e256 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:52.405984 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 157/310] 10.222.0.39 () {56 vars in 1308 bytes} [Fri Sep 30 09:26:52 2022] GET /identity/v3/role_assignments?user.id=9435e491ae594deab29b230ea7157a34&scope.system=all&role.id=05a780eae85c41adab97f133b17a6ed4 => generated 226 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:52.417042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2607ab67-9ad1-4d62-bc19-1e7741436c5f None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:52.417355 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2607ab67-9ad1-4d62-bc19-1e7741436c5f None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:52.417599 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2607ab67-9ad1-4d62-bc19-1e7741436c5f None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:52.419259 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 154/311] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:52.425748 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c5f93a-97f1-45d7-a815-0c7a6657484e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:52.426056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c5f93a-97f1-45d7-a815-0c7a6657484e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:52.426296 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c5f93a-97f1-45d7-a815-0c7a6657484e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:52.449718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-85c5f93a-97f1-45d7-a815-0c7a6657484e None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:52.450927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-85c5f93a-97f1-45d7-a815-0c7a6657484e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:52.534541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 158/312] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:52.805056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2607ab67-9ad1-4d62-bc19-1e7741436c5f None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:52.912999 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2607ab67-9ad1-4d62-bc19-1e7741436c5f None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:52.949877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22e0c53-fe0e-468b-bd05-e65de641a014 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:52.950527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22e0c53-fe0e-468b-bd05-e65de641a014 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:52.950841 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22e0c53-fe0e-468b-bd05-e65de641a014 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:52.951118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22e0c53-fe0e-468b-bd05-e65de641a014 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:52.962700 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22e0c53-fe0e-468b-bd05-e65de641a014 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:52.967705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-f22e0c53-fe0e-468b-bd05-e65de641a014 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 30 09:26:52.967705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:52.967705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:52.967705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:52.967705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:52.967705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:52.967705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:52.967705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:52.967705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:52.967705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:52.967705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:52.967705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:52.967705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:52.967705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:52.967705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:52.967705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:52.969536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:52.969536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:52.969536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:52.969536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:52.969536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:52.969536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:52.969536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:52.969536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:52.969536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:52.969536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:52.969536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:52.969536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:52.969536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:52.969536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:52.969536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:52.969536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:52.970353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:52.970353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:52.970353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:52.970353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:52.970353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:52.970353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:52.970353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:52.970353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:52.970353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:52.970353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:52.970353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 30 09:26:52.970353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:52.970353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 155/313] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:52 2022] GET /identity/v3/roles/reader => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:52.975254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c5f93a-97f1-45d7-a815-0c7a6657484e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:53.030359 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:53.079953 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c5f93a-97f1-45d7-a815-0c7a6657484e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:53.116476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c4adf1-cb96-4ce0-bebc-2ab5f551af5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:53.117374 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c4adf1-cb96-4ce0-bebc-2ab5f551af5e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:53.117851 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c4adf1-cb96-4ce0-bebc-2ab5f551af5e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:53.118293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c4adf1-cb96-4ce0-bebc-2ab5f551af5e None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:53.119195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c4adf1-cb96-4ce0-bebc-2ab5f551af5e None admin] RBAC: Adding query filter params (name=reader) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:53.119670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c4adf1-cb96-4ce0-bebc-2ab5f551af5e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:53.121205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c4adf1-cb96-4ce0-bebc-2ab5f551af5e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:53.130256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 159/314] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:52 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:53.136166 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22e0c53-fe0e-468b-bd05-e65de641a014 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:53.186978 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:53.251764 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22e0c53-fe0e-468b-bd05-e65de641a014 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:53.288512 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00e0045-60f9-4217-973b-a685618bb2bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:53.289498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00e0045-60f9-4217-973b-a685618bb2bf None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:53.289980 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00e0045-60f9-4217-973b-a685618bb2bf None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:53.290469 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00e0045-60f9-4217-973b-a685618bb2bf None admin] PATH_INFO: `/v3/users/24406c9ebcf448cbbe0517382694b044` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:53.291837 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00e0045-60f9-4217-973b-a685618bb2bf None admin] RBAC: Authorizing `identity:get_user(user_id=24406c9ebcf448cbbe0517382694b044)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:53.293556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00e0045-60f9-4217-973b-a685618bb2bf None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:53.295834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 156/315] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:53 2022] GET /identity/v3/users/24406c9ebcf448cbbe0517382694b044 => generated 297 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:53.301019 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c4adf1-cb96-4ce0-bebc-2ab5f551af5e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:53.410549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c4adf1-cb96-4ce0-bebc-2ab5f551af5e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:53.426706 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:53.444530 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69ac5b9-d98e-4f34-b565-c2750f3ecd3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:53.445130 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69ac5b9-d98e-4f34-b565-c2750f3ecd3d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:53.445406 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69ac5b9-d98e-4f34-b565-c2750f3ecd3d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:53.445674 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69ac5b9-d98e-4f34-b565-c2750f3ecd3d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:53.512553 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69ac5b9-d98e-4f34-b565-c2750f3ecd3d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:53.514062 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69ac5b9-d98e-4f34-b565-c2750f3ecd3d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:53.612038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 160/316] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 30 09:26:53.617424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00e0045-60f9-4217-973b-a685618bb2bf None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:53.726011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00e0045-60f9-4217-973b-a685618bb2bf None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:53.739842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:53.763130 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1438c6-8e07-4f66-9ac8-99575e392207 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:53.764137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1438c6-8e07-4f66-9ac8-99575e392207 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:53.764619 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1438c6-8e07-4f66-9ac8-99575e392207 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:53.765092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1438c6-8e07-4f66-9ac8-99575e392207 None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:53.767024 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1438c6-8e07-4f66-9ac8-99575e392207 None admin] RBAC: Authorizing `identity:get_project(project_id=25655c9cb779428f96b5430e993f1337)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:53.768744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1438c6-8e07-4f66-9ac8-99575e392207 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:53.771161 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 157/317] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:53 2022] GET /identity/v3/projects/25655c9cb779428f96b5430e993f1337 => generated 302 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:53.776492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69ac5b9-d98e-4f34-b565-c2750f3ecd3d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:53.793834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:53.875631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69ac5b9-d98e-4f34-b565-c2750f3ecd3d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:53.909877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957f9094-c1fd-4c5c-a69e-60cbc4093a1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:53.910480 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957f9094-c1fd-4c5c-a69e-60cbc4093a1a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:53.910784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957f9094-c1fd-4c5c-a69e-60cbc4093a1a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:53.911053 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957f9094-c1fd-4c5c-a69e-60cbc4093a1a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:53.911766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957f9094-c1fd-4c5c-a69e-60cbc4093a1a None admin] RBAC: Adding query filter params (role.id=05a780eae85c41adab97f133b17a6ed4, scope.project.id=25655c9cb779428f96b5430e993f1337, user.id=24406c9ebcf448cbbe0517382694b044) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:53.912069 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957f9094-c1fd-4c5c-a69e-60cbc4093a1a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:53.913558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957f9094-c1fd-4c5c-a69e-60cbc4093a1a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:53.919815 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 161/318] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:26:53 2022] GET /identity/v3/role_assignments?user.id=24406c9ebcf448cbbe0517382694b044&scope.project.id=25655c9cb779428f96b5430e993f1337&role.id=05a780eae85c41adab97f133b17a6ed4 => generated 608 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:54.296034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6e9ff4-c73e-48c9-9f6b-83e978d3e09f None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:54.297083 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6e9ff4-c73e-48c9-9f6b-83e978d3e09f None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:54.297471 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6e9ff4-c73e-48c9-9f6b-83e978d3e09f None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:54.300779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 158/319] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:54 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:54.305388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d530267b-de2a-41c6-9ee4-3324b35afb7c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:54.305833 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d530267b-de2a-41c6-9ee4-3324b35afb7c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:54.305998 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d530267b-de2a-41c6-9ee4-3324b35afb7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:54.323936 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d530267b-de2a-41c6-9ee4-3324b35afb7c None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:54.325106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d530267b-de2a-41c6-9ee4-3324b35afb7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:54.392173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 162/320] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:54.513474 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb265c25-ecba-434f-9861-44e3969922a2 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:54.513850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb265c25-ecba-434f-9861-44e3969922a2 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:54.514023 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb265c25-ecba-434f-9861-44e3969922a2 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:54.515560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 159/321] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:54.520680 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e351e8d3-20d5-454d-a087-8b76af3cc6bd None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:54.520968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e351e8d3-20d5-454d-a087-8b76af3cc6bd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:54.521241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e351e8d3-20d5-454d-a087-8b76af3cc6bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:54.541029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e351e8d3-20d5-454d-a087-8b76af3cc6bd None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:54.542272 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e351e8d3-20d5-454d-a087-8b76af3cc6bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:54.545584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:54.610641 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 163/322] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:54.646116 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98ad354-ba5d-4aa0-a32c-5e40b33c4610 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:54.646398 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98ad354-ba5d-4aa0-a32c-5e40b33c4610 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:54.646707 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98ad354-ba5d-4aa0-a32c-5e40b33c4610 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:54.648165 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 160/323] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:54.653029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c479f345-1b8b-4e0d-a6f4-156e805eb028 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:54.653326 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c479f345-1b8b-4e0d-a6f4-156e805eb028 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:54.653604 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c479f345-1b8b-4e0d-a6f4-156e805eb028 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:54.658692 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98ad354-ba5d-4aa0-a32c-5e40b33c4610 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:54.672837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c479f345-1b8b-4e0d-a6f4-156e805eb028 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:54.675047 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c479f345-1b8b-4e0d-a6f4-156e805eb028 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:54.754750 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 164/324] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:54.770556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98ad354-ba5d-4aa0-a32c-5e40b33c4610 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:54.803934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2cbb80-45c6-4eb3-a43c-62a67ddf51dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:54.804554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2cbb80-45c6-4eb3-a43c-62a67ddf51dd None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:54.804829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2cbb80-45c6-4eb3-a43c-62a67ddf51dd None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:54.805102 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2cbb80-45c6-4eb3-a43c-62a67ddf51dd None admin] PATH_INFO: `/v3/domains/default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:54.806184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2cbb80-45c6-4eb3-a43c-62a67ddf51dd None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:54.807652 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2cbb80-45c6-4eb3-a43c-62a67ddf51dd None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:54.809324 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 161/325] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:26:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:54.814776 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c479f345-1b8b-4e0d-a6f4-156e805eb028 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:54.876721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2cbb80-45c6-4eb3-a43c-62a67ddf51dd None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:54.882715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:54.915893 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c479f345-1b8b-4e0d-a6f4-156e805eb028 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:54.928193 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:54.951801 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53de173-0991-4176-b077-2314ea094fba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:54.952685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53de173-0991-4176-b077-2314ea094fba None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:54.953509 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53de173-0991-4176-b077-2314ea094fba None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:54.953784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53de173-0991-4176-b077-2314ea094fba None admin] PATH_INFO: `/v3/groups` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:54.954417 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53de173-0991-4176-b077-2314ea094fba None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:54.955915 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53de173-0991-4176-b077-2314ea094fba None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:54.984993 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2cbb80-45c6-4eb3-a43c-62a67ddf51dd None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:55.019381 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6d7817-44cb-4644-8f65-b9d22bb49968 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:55.020004 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6d7817-44cb-4644-8f65-b9d22bb49968 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:55.020300 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6d7817-44cb-4644-8f65-b9d22bb49968 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:55.020550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6d7817-44cb-4644-8f65-b9d22bb49968 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:55.032466 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6d7817-44cb-4644-8f65-b9d22bb49968 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:55.035885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-2b6d7817-44cb-4644-8f65-b9d22bb49968 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:55.035885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:55.035885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:55.035885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:55.035885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:55.035885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:55.035885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:55.035885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:55.035885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:55.035885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:55.035885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:55.035885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:55.035885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:55.035885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:55.035885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:55.035885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:55.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:55.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:55.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:55.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:55.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:55.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:55.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:55.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:55.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:55.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:55.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:55.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:55.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:55.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:55.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:55.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:55.037855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:55.037855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:55.037855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:55.037855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:55.037855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:55.037855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:55.037855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:55.037855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:55.037855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:55.037855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:55.037855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:55.037855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:55.037855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 162/326] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:26:55.037855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6d7817-44cb-4644-8f65-b9d22bb49968 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:55.056296 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 165/327] 10.222.0.39 () {60 vars in 1134 bytes} [Fri Sep 30 09:26:54 2022] POST /identity/v3/groups => generated 226 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:26:55.057375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53de173-0991-4176-b077-2314ea094fba None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:55.151860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6d7817-44cb-4644-8f65-b9d22bb49968 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:55.162201 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:55.174486 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53de173-0991-4176-b077-2314ea094fba None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:55.193190 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a322ee4-3211-4374-bcfd-f462bbcbbc34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:55.196343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a322ee4-3211-4374-bcfd-f462bbcbbc34 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:55.196343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a322ee4-3211-4374-bcfd-f462bbcbbc34 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:55.196343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a322ee4-3211-4374-bcfd-f462bbcbbc34 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:55.207738 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a322ee4-3211-4374-bcfd-f462bbcbbc34 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:55.211505 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:55.212954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-5a322ee4-3211-4374-bcfd-f462bbcbbc34 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 30 09:26:55.212954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:55.212954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:55.212954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:55.212954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:55.212954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:55.212954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:55.212954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:55.212954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:55.212954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:55.212954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:55.212954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:55.212954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:55.212954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:55.212954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:55.212954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:55.213856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:55.213856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:55.213856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:55.213856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:55.213856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:55.213856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:55.213856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:55.213856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:55.213856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:55.213856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:55.213856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:55.213856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:55.213856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:55.213856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:55.213856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:55.213856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:55.214702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:55.214702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:55.214702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:55.214702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:55.214702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:55.214702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:55.214702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:55.214702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:55.214702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:55.214702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:55.214702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 30 09:26:55.214702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:55.214702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 163/328] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:55 2022] GET /identity/v3/roles/reader => generated 84 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:55.220133 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a322ee4-3211-4374-bcfd-f462bbcbbc34 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:55.220643 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177ed923-21b2-4687-b360-dd17a7604c67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:55.221342 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177ed923-21b2-4687-b360-dd17a7604c67 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:55.221691 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177ed923-21b2-4687-b360-dd17a7604c67 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:55.222049 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177ed923-21b2-4687-b360-dd17a7604c67 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:55.222952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177ed923-21b2-4687-b360-dd17a7604c67 None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:55.232631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177ed923-21b2-4687-b360-dd17a7604c67 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:55.232631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177ed923-21b2-4687-b360-dd17a7604c67 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:55.234605 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 166/329] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:55 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:55.240725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177ed923-21b2-4687-b360-dd17a7604c67 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:55.343326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a322ee4-3211-4374-bcfd-f462bbcbbc34 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:55.359787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177ed923-21b2-4687-b360-dd17a7604c67 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:55.381926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff2cdca-eec5-41d0-9280-ffb667dbfe7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:55.382829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff2cdca-eec5-41d0-9280-ffb667dbfe7f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:55.383109 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff2cdca-eec5-41d0-9280-ffb667dbfe7f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:55.383381 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff2cdca-eec5-41d0-9280-ffb667dbfe7f None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:55.384049 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff2cdca-eec5-41d0-9280-ffb667dbfe7f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:55.384350 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff2cdca-eec5-41d0-9280-ffb667dbfe7f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:55.385869 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff2cdca-eec5-41d0-9280-ffb667dbfe7f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:55.395786 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 164/330] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:55 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:55.401603 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff2cdca-eec5-41d0-9280-ffb667dbfe7f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:55.403477 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ff9c1b-3942-4931-bd53-c38b17e3557b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:55.404141 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ff9c1b-3942-4931-bd53-c38b17e3557b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:55.404419 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ff9c1b-3942-4931-bd53-c38b17e3557b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:55.404877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ff9c1b-3942-4931-bd53-c38b17e3557b None admin] PATH_INFO: `/v3/users/41eca216f4d3472cae8a134cd4ee80ed` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:55.406066 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ff9c1b-3942-4931-bd53-c38b17e3557b None admin] RBAC: Authorizing `identity:get_user(user_id=41eca216f4d3472cae8a134cd4ee80ed)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:55.408458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ff9c1b-3942-4931-bd53-c38b17e3557b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:55.410740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 167/331] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:55 2022] GET /identity/v3/users/41eca216f4d3472cae8a134cd4ee80ed => generated 293 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:55.415649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ff9c1b-3942-4931-bd53-c38b17e3557b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:55.445588 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:55.506006 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:55.547025 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff2cdca-eec5-41d0-9280-ffb667dbfe7f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:55.564838 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ff9c1b-3942-4931-bd53-c38b17e3557b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:55.588913 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9761a368-0887-4793-84ef-6af666270aa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:55.589893 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9761a368-0887-4793-84ef-6af666270aa5 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:55.590322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9761a368-0887-4793-84ef-6af666270aa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:55.590772 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9761a368-0887-4793-84ef-6af666270aa5 None admin] PATH_INFO: `/v3/users/9435e491ae594deab29b230ea7157a34` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:55.592214 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9761a368-0887-4793-84ef-6af666270aa5 None admin] RBAC: Authorizing `identity:get_user(user_id=9435e491ae594deab29b230ea7157a34)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:55.594017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9761a368-0887-4793-84ef-6af666270aa5 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:55.596176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 165/332] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:26:55 2022] GET /identity/v3/users/9435e491ae594deab29b230ea7157a34 => generated 293 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:55.604086 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9761a368-0887-4793-84ef-6af666270aa5 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:55.608210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78fe952e-baf5-4f58-a027-763efb6674fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:55.610316 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fe952e-baf5-4f58-a027-763efb6674fc None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:55.610632 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fe952e-baf5-4f58-a027-763efb6674fc None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:55.610915 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fe952e-baf5-4f58-a027-763efb6674fc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:55.611643 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78fe952e-baf5-4f58-a027-763efb6674fc None admin] RBAC: Adding query filter params (role.id=025f58735650469d9fa6faca3e9e7998, user.id=41eca216f4d3472cae8a134cd4ee80ed, scope.system=all) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:55.611951 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78fe952e-baf5-4f58-a027-763efb6674fc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:55.613387 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78fe952e-baf5-4f58-a027-763efb6674fc None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:55.629603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 168/333] 10.222.0.39 () {56 vars in 1308 bytes} [Fri Sep 30 09:26:55 2022] GET /identity/v3/role_assignments?user.id=41eca216f4d3472cae8a134cd4ee80ed&scope.system=all&role.id=025f58735650469d9fa6faca3e9e7998 => generated 510 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:26:55.733417 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:55.739880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9761a368-0887-4793-84ef-6af666270aa5 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:55.778696 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc41ec3e-feec-4194-b75d-e74f50d28d56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:55.779367 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc41ec3e-feec-4194-b75d-e74f50d28d56 None admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:55.779644 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc41ec3e-feec-4194-b75d-e74f50d28d56 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:55.779978 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc41ec3e-feec-4194-b75d-e74f50d28d56 None admin] PATH_INFO: `/v3/system/users/9435e491ae594deab29b230ea7157a34/roles/05a780eae85c41adab97f133b17a6ed4` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:55.781967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc41ec3e-feec-4194-b75d-e74f50d28d56 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=9435e491ae594deab29b230ea7157a34, role_id=05a780eae85c41adab97f133b17a6ed4)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:55.783422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc41ec3e-feec-4194-b75d-e74f50d28d56 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:55.851743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 166/334] 10.222.0.39 () {58 vars in 1334 bytes} [Fri Sep 30 09:26:55 2022] PUT /identity/v3/system/users/9435e491ae594deab29b230ea7157a34/roles/05a780eae85c41adab97f133b17a6ed4 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:26:57.654672 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912271fe-0e6a-47a4-8f73-c49bb5fdc460 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:57.655214 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912271fe-0e6a-47a4-8f73-c49bb5fdc460 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:57.655214 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912271fe-0e6a-47a4-8f73-c49bb5fdc460 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:57.657419 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 169/335] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:57.664015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015a979d-ac88-4d87-a882-8812a351dbcd None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:57.664460 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015a979d-ac88-4d87-a882-8812a351dbcd None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:57.664547 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015a979d-ac88-4d87-a882-8812a351dbcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:57.691566 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-015a979d-ac88-4d87-a882-8812a351dbcd None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:57.692778 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-015a979d-ac88-4d87-a882-8812a351dbcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:57.762625 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 167/336] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:58.320798 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912271fe-0e6a-47a4-8f73-c49bb5fdc460 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:58.391054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e631fd-9055-4d23-a06e-92172094e5bb None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:58.391817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e631fd-9055-4d23-a06e-92172094e5bb None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:58.392647 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e631fd-9055-4d23-a06e-92172094e5bb None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:58.394811 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 168/337] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:58 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:58.400403 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23d9d07-cb3c-4185-9308-9dce287b7ddb None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:58.401119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23d9d07-cb3c-4185-9308-9dce287b7ddb None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:58.401632 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23d9d07-cb3c-4185-9308-9dce287b7ddb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:58.436549 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-c23d9d07-cb3c-4185-9308-9dce287b7ddb None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:58.437811 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c23d9d07-cb3c-4185-9308-9dce287b7ddb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:58.458568 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912271fe-0e6a-47a4-8f73-c49bb5fdc460 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:58.493862 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:58.500435 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39cbbc9d-f466-47fd-a879-ab7b04668e27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:58.501078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cbbc9d-f466-47fd-a879-ab7b04668e27 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:58.504469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cbbc9d-f466-47fd-a879-ab7b04668e27 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:58.504469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cbbc9d-f466-47fd-a879-ab7b04668e27 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:58.529942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39cbbc9d-f466-47fd-a879-ab7b04668e27 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:58.535116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-39cbbc9d-f466-47fd-a879-ab7b04668e27 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:58.535116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:58.535116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:58.535116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:58.535116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:58.535116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:58.535116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:58.535116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:58.535116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:58.535116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:58.535116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:58.535116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:58.535116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:58.535116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:58.535116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:58.535116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:58.536070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:58.536070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:58.536070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:58.536070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:58.536070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:58.536070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:58.536070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:58.536070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:58.536070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:58.536070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:58.536070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:58.536070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:58.536070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:58.536070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:58.536070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:58.536070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:58.537349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:58.537349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:58.537349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:58.537349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:58.537349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:58.537349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:58.537349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:58.537349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:58.537349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:58.537349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:58.537349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:58.537349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:26:58.537349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 170/338] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:58 2022] GET /identity/v3/roles/member => generated 84 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:58.538623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295a9204-979b-44ce-9e35-85e199138cac None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:58.538623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295a9204-979b-44ce-9e35-85e199138cac None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:58.540584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295a9204-979b-44ce-9e35-85e199138cac None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:58.540584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 171/339] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:58.546889 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c3592-0178-4917-8792-e6fe1b893102 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:58.547360 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c3592-0178-4917-8792-e6fe1b893102 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:58.547620 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c3592-0178-4917-8792-e6fe1b893102 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:58.570729 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 169/340] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:58.579620 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23d9d07-cb3c-4185-9308-9dce287b7ddb None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:58.592421 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-946c3592-0178-4917-8792-e6fe1b893102 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:58.599094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-946c3592-0178-4917-8792-e6fe1b893102 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:58.644651 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:58.758441 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 172/341] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:26:58.784925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23d9d07-cb3c-4185-9308-9dce287b7ddb None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:58.823653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095de5d4-f76b-451c-923a-a3269a29454f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:58.824267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095de5d4-f76b-451c-923a-a3269a29454f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:58.824547 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095de5d4-f76b-451c-923a-a3269a29454f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:58.824818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095de5d4-f76b-451c-923a-a3269a29454f None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:58.825481 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095de5d4-f76b-451c-923a-a3269a29454f None admin] RBAC: Adding query filter params (name=member) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:58.825785 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095de5d4-f76b-451c-923a-a3269a29454f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:58.827203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095de5d4-f76b-451c-923a-a3269a29454f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:58.835977 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 170/342] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:58 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:58.841695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946c3592-0178-4917-8792-e6fe1b893102 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:58.854149 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be8fb7f-7ede-4291-abcb-1c1def1746f9 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:58.854433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be8fb7f-7ede-4291-abcb-1c1def1746f9 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:58.856333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be8fb7f-7ede-4291-abcb-1c1def1746f9 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:58.857670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 171/343] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:58.864222 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2ba09c-8d5a-44ea-8453-e8fb41f2e4cd None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:58.864504 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2ba09c-8d5a-44ea-8453-e8fb41f2e4cd None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:58.864778 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2ba09c-8d5a-44ea-8453-e8fb41f2e4cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:58.903686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-bc2ba09c-8d5a-44ea-8453-e8fb41f2e4cd None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:26:58.906372 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-bc2ba09c-8d5a-44ea-8453-e8fb41f2e4cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:26:59.016971 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 172/344] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:26:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:26:59.050717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946c3592-0178-4917-8792-e6fe1b893102 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:59.068531 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:59.104148 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01440ee-992c-408e-b564-7053dcf17c2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:59.104769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01440ee-992c-408e-b564-7053dcf17c2c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:59.105046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01440ee-992c-408e-b564-7053dcf17c2c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:59.105316 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01440ee-992c-408e-b564-7053dcf17c2c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:59.126321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2ba09c-8d5a-44ea-8453-e8fb41f2e4cd None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:59.163050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:59.267941 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01440ee-992c-408e-b564-7053dcf17c2c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:59.269330 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01440ee-992c-408e-b564-7053dcf17c2c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:59.310422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2ba09c-8d5a-44ea-8453-e8fb41f2e4cd None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:59.354190 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5464c499-23c2-4e59-a32f-f22252aee806 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:59.354923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5464c499-23c2-4e59-a32f-f22252aee806 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:59.355204 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5464c499-23c2-4e59-a32f-f22252aee806 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:59.355477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5464c499-23c2-4e59-a32f-f22252aee806 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:59.368519 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5464c499-23c2-4e59-a32f-f22252aee806 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:59.374294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-5464c499-23c2-4e59-a32f-f22252aee806 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:59.374294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:59.374294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:59.374294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:59.374294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:59.374294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:59.374294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:59.374294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:59.374294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:59.374294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:59.374294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:59.374294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:59.374294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:59.374294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:59.374294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:59.374294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:59.375357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:59.375357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:59.375357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:59.375357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:59.375357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:59.375357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:59.375357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:59.375357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:59.375357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:59.375357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:59.375357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:59.375357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:59.375357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:59.375357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:59.375357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:59.375357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:59.376159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:59.376159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:59.376159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:59.376159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:59.376159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:59.376159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:59.376159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:59.376159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:59.376159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:59.376159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:59.376159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:26:59.376159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:59.377327 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 173/345] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:26:59.377394 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5464c499-23c2-4e59-a32f-f22252aee806 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:59.468776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:59.484004 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 173/346] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 643 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:26:59.484867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01440ee-992c-408e-b564-7053dcf17c2c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:59.501638 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:59.511468 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5464c499-23c2-4e59-a32f-f22252aee806 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:59.557884 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf46b4b-b29a-4c56-bfe4-766414a79c2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:59.558436 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf46b4b-b29a-4c56-bfe4-766414a79c2e None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:59.558804 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf46b4b-b29a-4c56-bfe4-766414a79c2e None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:59.559076 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf46b4b-b29a-4c56-bfe4-766414a79c2e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:59.572477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf46b4b-b29a-4c56-bfe4-766414a79c2e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:59.579199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-bcf46b4b-b29a-4c56-bfe4-766414a79c2e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:26:59.579199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:59.579199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:59.579199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:59.579199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:59.579199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:59.579199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:59.579199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:59.579199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:59.579199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:59.579199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:59.579199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:59.579199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:59.579199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:59.579199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:59.579199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:59.580220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:59.580220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:59.580220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:59.580220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:59.580220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:59.580220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:59.580220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:59.580220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:59.580220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:59.580220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:59.580220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:59.580220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:59.580220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:59.580220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:59.580220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:59.580220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:59.581056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:59.581056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:59.581056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:59.581056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:59.581056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:59.581056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:59.581056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:59.581056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:59.581056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:59.581056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:59.581056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:26:59.581056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:26:59.581056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 174/347] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:26:59 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:59.581729 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf46b4b-b29a-4c56-bfe4-766414a79c2e None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:59.624941 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01440ee-992c-408e-b564-7053dcf17c2c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:59.665944 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d4405f-3a73-448f-acd7-e8daf121e821 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:59.666942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d4405f-3a73-448f-acd7-e8daf121e821 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:59.667427 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d4405f-3a73-448f-acd7-e8daf121e821 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:59.667898 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d4405f-3a73-448f-acd7-e8daf121e821 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:59.668755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d4405f-3a73-448f-acd7-e8daf121e821 None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:59.669255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d4405f-3a73-448f-acd7-e8daf121e821 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:59.671943 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d4405f-3a73-448f-acd7-e8daf121e821 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:59.682665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 174/348] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:26:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:59.683845 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d4405f-3a73-448f-acd7-e8daf121e821 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:59.710378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf46b4b-b29a-4c56-bfe4-766414a79c2e None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:59.747324 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a363c5-2d2d-4c24-8e40-0b36ef3a959a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:59.747966 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a363c5-2d2d-4c24-8e40-0b36ef3a959a None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:59.748251 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a363c5-2d2d-4c24-8e40-0b36ef3a959a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:59.748529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a363c5-2d2d-4c24-8e40-0b36ef3a959a None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:59.750239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a363c5-2d2d-4c24-8e40-0b36ef3a959a None admin] RBAC: Authorizing `identity:get_project(project_id=333c93c6a3ef48b6806ea0d5f14ce211)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:59.751835 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a363c5-2d2d-4c24-8e40-0b36ef3a959a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:59.753527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 175/349] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:26:59 2022] GET /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211 => generated 298 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:59.754466 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9de20be-d62f-4fa5-aaa9-fb621fc39622 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:59.754792 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9de20be-d62f-4fa5-aaa9-fb621fc39622 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:59.755067 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9de20be-d62f-4fa5-aaa9-fb621fc39622 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:59.756503 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 176/350] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:26:59 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:26:59.757446 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9de20be-d62f-4fa5-aaa9-fb621fc39622 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:59.760284 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:59.780310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:26:59.816036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d4405f-3a73-448f-acd7-e8daf121e821 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:59.864496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4c145d-6e55-46d2-8027-467ecc04593f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:59.865138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4c145d-6e55-46d2-8027-467ecc04593f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:59.865418 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4c145d-6e55-46d2-8027-467ecc04593f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:59.866840 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4c145d-6e55-46d2-8027-467ecc04593f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:59.884013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4c145d-6e55-46d2-8027-467ecc04593f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:26:59.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-0c4c145d-6e55-46d2-8027-467ecc04593f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 30 09:26:59.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:26:59.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:26:59.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:26:59.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:26:59.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:26:59.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:26:59.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:26:59.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:26:59.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:26:59.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:26:59.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:26:59.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:26:59.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:26:59.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:26:59.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:26:59.891225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:26:59.891225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:26:59.891225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:26:59.891225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:26:59.891225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:26:59.891225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:26:59.891225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:26:59.891225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:26:59.891225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:26:59.891225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:26:59.891225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:26:59.891225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:26:59.891225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:26:59.891225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:26:59.891225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:26:59.891225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:26:59.892031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:26:59.892031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:26:59.892031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:26:59.892031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:26:59.892031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:26:59.892031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:26:59.892031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:26:59.892031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:26:59.892031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:26:59.892031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:26:59.892031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 30 09:26:59.892031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:26:59.892670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9de20be-d62f-4fa5-aaa9-fb621fc39622 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:26:59.893066 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 175/351] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:26:59 2022] GET /identity/v3/roles/reader => generated 84 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:26:59.893066 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4c145d-6e55-46d2-8027-467ecc04593f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:26:59.931257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9295bd6-7990-40c2-a793-db25060bf9ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:26:59.931937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9295bd6-7990-40c2-a793-db25060bf9ad None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:26:59.932218 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9295bd6-7990-40c2-a793-db25060bf9ad None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:26:59.932490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9295bd6-7990-40c2-a793-db25060bf9ad None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:26:59.933215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9295bd6-7990-40c2-a793-db25060bf9ad None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:26:59.933515 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9295bd6-7990-40c2-a793-db25060bf9ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:26:59.935478 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9295bd6-7990-40c2-a793-db25060bf9ad None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:26:59.947309 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 177/352] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:26:59 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:26:59.948140 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9295bd6-7990-40c2-a793-db25060bf9ad None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:00.015438 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4c145d-6e55-46d2-8027-467ecc04593f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:00.036142 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:00.057621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:00.072722 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc4ccde-50b3-44b2-82d9-96e0af750124 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:00.072722 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc4ccde-50b3-44b2-82d9-96e0af750124 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:00.072722 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc4ccde-50b3-44b2-82d9-96e0af750124 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:00.072722 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc4ccde-50b3-44b2-82d9-96e0af750124 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:00.084166 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9295bd6-7990-40c2-a793-db25060bf9ad None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:00.126635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6121dd-bdfa-4fd2-8aa9-495dd0e600f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:00.126635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6121dd-bdfa-4fd2-8aa9-495dd0e600f0 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:00.126930 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6121dd-bdfa-4fd2-8aa9-495dd0e600f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:00.127211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6121dd-bdfa-4fd2-8aa9-495dd0e600f0 None admin] PATH_INFO: `/v3/groups/819dc62c48e64e6a97d7337cb2595c5d` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:00.135100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6121dd-bdfa-4fd2-8aa9-495dd0e600f0 None admin] RBAC: Authorizing `identity:get_group(group_id=819dc62c48e64e6a97d7337cb2595c5d)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:00.136564 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6121dd-bdfa-4fd2-8aa9-495dd0e600f0 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:00.138319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 178/353] 10.222.0.39 () {56 vars in 1180 bytes} [Fri Sep 30 09:26:59 2022] GET /identity/v3/groups/819dc62c48e64e6a97d7337cb2595c5d => generated 226 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:00.139472 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1232e20-5f41-43c2-a5e7-8a6f0ee8ffd9 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:00.139757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1232e20-5f41-43c2-a5e7-8a6f0ee8ffd9 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:00.140059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1232e20-5f41-43c2-a5e7-8a6f0ee8ffd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:00.151294 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc4ccde-50b3-44b2-82d9-96e0af750124 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:00.152775 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc4ccde-50b3-44b2-82d9-96e0af750124 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:00.159959 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-c1232e20-5f41-43c2-a5e7-8a6f0ee8ffd9 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:00.161149 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c1232e20-5f41-43c2-a5e7-8a6f0ee8ffd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:00.266723 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 179/354] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:00.266723 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1232e20-5f41-43c2-a5e7-8a6f0ee8ffd9 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:00.289510 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 176/355] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:26:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:27:00.296819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc4ccde-50b3-44b2-82d9-96e0af750124 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:00.326754 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:00.340401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:00.386305 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1232e20-5f41-43c2-a5e7-8a6f0ee8ffd9 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:00.433738 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e09cb8a-e351-4eb7-85f6-5a6010ffa4a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:00.434359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e09cb8a-e351-4eb7-85f6-5a6010ffa4a4 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:00.435481 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e09cb8a-e351-4eb7-85f6-5a6010ffa4a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:00.435481 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e09cb8a-e351-4eb7-85f6-5a6010ffa4a4 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:00.435658 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc4ccde-50b3-44b2-82d9-96e0af750124 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:00.435746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e09cb8a-e351-4eb7-85f6-5a6010ffa4a4 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:00.435928 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e09cb8a-e351-4eb7-85f6-5a6010ffa4a4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:00.437358 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e09cb8a-e351-4eb7-85f6-5a6010ffa4a4 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:00.446560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 180/356] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:27:00 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:00.447667 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de47cba3-b88a-4158-8664-c044c674634b None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:00.447954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de47cba3-b88a-4158-8664-c044c674634b None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:00.448168 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de47cba3-b88a-4158-8664-c044c674634b None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:00.449354 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 181/357] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:00 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:00.450036 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de47cba3-b88a-4158-8664-c044c674634b None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:00.486560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769b2ab5-ef1f-43c2-a9bd-bfe5a94c8964 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:00.487221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769b2ab5-ef1f-43c2-a9bd-bfe5a94c8964 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:00.487701 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769b2ab5-ef1f-43c2-a9bd-bfe5a94c8964 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:00.487988 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769b2ab5-ef1f-43c2-a9bd-bfe5a94c8964 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:00.574712 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769b2ab5-ef1f-43c2-a9bd-bfe5a94c8964 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:00.575479 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de47cba3-b88a-4158-8664-c044c674634b None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:00.576178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769b2ab5-ef1f-43c2-a9bd-bfe5a94c8964 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:00.600912 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:00.611658 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:00.616666 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b82e559-3912-4eaa-aa91-de68d1a4b413 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:00.617271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b82e559-3912-4eaa-aa91-de68d1a4b413 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:00.617563 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b82e559-3912-4eaa-aa91-de68d1a4b413 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:00.618564 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b82e559-3912-4eaa-aa91-de68d1a4b413 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:00.618564 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b82e559-3912-4eaa-aa91-de68d1a4b413 None admin] RBAC: Adding query filter params (group.id=819dc62c48e64e6a97d7337cb2595c5d, role.id=025f58735650469d9fa6faca3e9e7998, scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:00.618877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b82e559-3912-4eaa-aa91-de68d1a4b413 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:00.620309 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b82e559-3912-4eaa-aa91-de68d1a4b413 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:00.627167 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 182/358] 10.222.0.39 () {56 vars in 1376 bytes} [Fri Sep 30 09:27:00 2022] GET /identity/v3/role_assignments?group.id=819dc62c48e64e6a97d7337cb2595c5d&scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211&role.id=025f58735650469d9fa6faca3e9e7998 => generated 260 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:00.627757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b82e559-3912-4eaa-aa91-de68d1a4b413 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:00.708129 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 177/359] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:27:00.709588 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f5cc20-4652-41f5-b68e-50b129990f8d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:00.711384 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f5cc20-4652-41f5-b68e-50b129990f8d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:00.711384 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f5cc20-4652-41f5-b68e-50b129990f8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:00.732086 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c5f5cc20-4652-41f5-b68e-50b129990f8d None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:00.733355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c5f5cc20-4652-41f5-b68e-50b129990f8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:00.769646 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b82e559-3912-4eaa-aa91-de68d1a4b413 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:00.821577 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4a6c2a-2fbd-4864-811e-4129df06bfa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:00.822221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4a6c2a-2fbd-4864-811e-4129df06bfa7 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:00.822504 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4a6c2a-2fbd-4864-811e-4129df06bfa7 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:00.823666 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 178/360] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:00.824119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4a6c2a-2fbd-4864-811e-4129df06bfa7 None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:00.824513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f5cc20-4652-41f5-b68e-50b129990f8d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:00.824913 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4a6c2a-2fbd-4864-811e-4129df06bfa7 None admin] RBAC: Authorizing `identity:get_project(project_id=25655c9cb779428f96b5430e993f1337)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:00.826482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4a6c2a-2fbd-4864-811e-4129df06bfa7 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:00.828695 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 183/361] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:27:00 2022] GET /identity/v3/projects/25655c9cb779428f96b5430e993f1337 => generated 302 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:00.829458 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4a6c2a-2fbd-4864-811e-4129df06bfa7 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:00.898138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:00.907744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:00.961055 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f5cc20-4652-41f5-b68e-50b129990f8d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:00.972153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4a6c2a-2fbd-4864-811e-4129df06bfa7 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:01.014647 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b99428b-6ebd-4082-8f00-cec74f0fe7fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:01.015843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b99428b-6ebd-4082-8f00-cec74f0fe7fd None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:01.016120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b99428b-6ebd-4082-8f00-cec74f0fe7fd None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:01.016389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b99428b-6ebd-4082-8f00-cec74f0fe7fd None admin] PATH_INFO: `/v3/users/9435e491ae594deab29b230ea7157a34` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:01.017801 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b99428b-6ebd-4082-8f00-cec74f0fe7fd None admin] RBAC: Authorizing `identity:get_user(user_id=9435e491ae594deab29b230ea7157a34)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:01.019366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b99428b-6ebd-4082-8f00-cec74f0fe7fd None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:01.020189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25af017-ca80-4716-9305-8856a0455dfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:01.021124 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25af017-ca80-4716-9305-8856a0455dfc None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:01.021364 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 179/362] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:27:00 2022] GET /identity/v3/users/9435e491ae594deab29b230ea7157a34 => generated 293 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:01.021956 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25af017-ca80-4716-9305-8856a0455dfc None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:01.022061 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b99428b-6ebd-4082-8f00-cec74f0fe7fd None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:01.022544 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25af017-ca80-4716-9305-8856a0455dfc None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:01.034472 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25af017-ca80-4716-9305-8856a0455dfc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:01.041034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-a25af017-ca80-4716-9305-8856a0455dfc None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:27:01.041034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:01.041034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:01.041034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:01.041034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:01.041034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:01.041034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:01.041034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:01.041034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:01.041034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:01.041034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:01.041034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:01.041034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:01.041034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:01.041034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:01.041034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:01.041998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:01.041998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:01.041998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:01.041998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:01.041998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:01.041998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:01.041998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:01.041998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:01.041998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:01.041998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:01.041998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:01.041998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:01.041998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:01.041998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:01.041998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:01.041998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:01.042888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:01.042888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:01.042888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:01.042888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:01.042888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:01.042888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:01.042888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:01.042888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:01.042888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:01.042888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:01.042888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:27:01.042888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:01.045189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 184/363] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:27:00 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:01.046313 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25af017-ca80-4716-9305-8856a0455dfc None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:01.136888 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b99428b-6ebd-4082-8f00-cec74f0fe7fd None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:01.163111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25af017-ca80-4716-9305-8856a0455dfc None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:01.174825 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:01.181137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:01.189009 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8365ed16-ab8b-4bfc-a9ab-df921297fca8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:01.194188 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8365ed16-ab8b-4bfc-a9ab-df921297fca8 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:01.194188 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8365ed16-ab8b-4bfc-a9ab-df921297fca8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:01.194188 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8365ed16-ab8b-4bfc-a9ab-df921297fca8 None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:01.194188 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8365ed16-ab8b-4bfc-a9ab-df921297fca8 None admin] RBAC: Authorizing `identity:get_project(project_id=333c93c6a3ef48b6806ea0d5f14ce211)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:01.194188 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8365ed16-ab8b-4bfc-a9ab-df921297fca8 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:01.195670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 180/364] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:27:01 2022] GET /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211 => generated 298 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:01.196476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8365ed16-ab8b-4bfc-a9ab-df921297fca8 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:01.208617 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651f0042-3295-49a1-a1ae-c6f92582d481 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:01.209273 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651f0042-3295-49a1-a1ae-c6f92582d481 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:01.209779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651f0042-3295-49a1-a1ae-c6f92582d481 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:01.209942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651f0042-3295-49a1-a1ae-c6f92582d481 None admin] PATH_INFO: `/v3/groups/819dc62c48e64e6a97d7337cb2595c5d` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:01.211801 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651f0042-3295-49a1-a1ae-c6f92582d481 None admin] RBAC: Authorizing `identity:get_group(group_id=819dc62c48e64e6a97d7337cb2595c5d)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:01.213285 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651f0042-3295-49a1-a1ae-c6f92582d481 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:01.215071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 185/365] 10.222.0.39 () {56 vars in 1180 bytes} [Fri Sep 30 09:27:01 2022] GET /identity/v3/groups/819dc62c48e64e6a97d7337cb2595c5d => generated 226 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:01.215837 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651f0042-3295-49a1-a1ae-c6f92582d481 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:01.333944 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8365ed16-ab8b-4bfc-a9ab-df921297fca8 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:01.353594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651f0042-3295-49a1-a1ae-c6f92582d481 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:01.381130 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb7bb23-12d3-4db5-a1ea-f8bd852be269 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:01.382436 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb7bb23-12d3-4db5-a1ea-f8bd852be269 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:01.382915 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb7bb23-12d3-4db5-a1ea-f8bd852be269 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:01.383359 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb7bb23-12d3-4db5-a1ea-f8bd852be269 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:01.384924 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb7bb23-12d3-4db5-a1ea-f8bd852be269 None admin] RBAC: Adding query filter params (role.id=05a780eae85c41adab97f133b17a6ed4, user.id=9435e491ae594deab29b230ea7157a34, scope.system=all) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:01.385401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb7bb23-12d3-4db5-a1ea-f8bd852be269 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:01.388419 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb7bb23-12d3-4db5-a1ea-f8bd852be269 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:01.400811 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e44c39-fbdb-4fac-96dd-9ab79ed49c1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:01.401433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e44c39-fbdb-4fac-96dd-9ab79ed49c1b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:01.402179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e44c39-fbdb-4fac-96dd-9ab79ed49c1b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:01.402459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e44c39-fbdb-4fac-96dd-9ab79ed49c1b None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:01.403181 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e44c39-fbdb-4fac-96dd-9ab79ed49c1b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:01.403491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e44c39-fbdb-4fac-96dd-9ab79ed49c1b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:01.404886 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e44c39-fbdb-4fac-96dd-9ab79ed49c1b None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:01.409276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 181/366] 10.222.0.39 () {56 vars in 1308 bytes} [Fri Sep 30 09:27:01 2022] GET /identity/v3/role_assignments?user.id=9435e491ae594deab29b230ea7157a34&scope.system=all&role.id=05a780eae85c41adab97f133b17a6ed4 => generated 510 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:01.409276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb7bb23-12d3-4db5-a1ea-f8bd852be269 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:01.423627 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 186/367] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:01 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:01.425157 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e44c39-fbdb-4fac-96dd-9ab79ed49c1b None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:01.479347 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:01.490576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:01.553540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e44c39-fbdb-4fac-96dd-9ab79ed49c1b None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:01.564882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb7bb23-12d3-4db5-a1ea-f8bd852be269 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:01.596631 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc748d9b-22cf-42f5-bb5b-9d4b7b8bf6c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:01.597518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc748d9b-22cf-42f5-bb5b-9d4b7b8bf6c2 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:01.597979 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc748d9b-22cf-42f5-bb5b-9d4b7b8bf6c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:01.598407 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc748d9b-22cf-42f5-bb5b-9d4b7b8bf6c2 None admin] PATH_INFO: `/v3/groups/819dc62c48e64e6a97d7337cb2595c5d` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:01.599790 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc748d9b-22cf-42f5-bb5b-9d4b7b8bf6c2 None admin] RBAC: Authorizing `identity:get_group(group_id=819dc62c48e64e6a97d7337cb2595c5d)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:01.601515 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc748d9b-22cf-42f5-bb5b-9d4b7b8bf6c2 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:01.603468 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 187/368] 10.222.0.39 () {56 vars in 1180 bytes} [Fri Sep 30 09:27:01 2022] GET /identity/v3/groups/819dc62c48e64e6a97d7337cb2595c5d => generated 226 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:01.604482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc748d9b-22cf-42f5-bb5b-9d4b7b8bf6c2 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:01.609910 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae24efc-f362-4881-8393-8efd6e71944d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:01.609910 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae24efc-f362-4881-8393-8efd6e71944d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:01.610169 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae24efc-f362-4881-8393-8efd6e71944d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:01.610463 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae24efc-f362-4881-8393-8efd6e71944d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:01.622760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae24efc-f362-4881-8393-8efd6e71944d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:01.629488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-7ae24efc-f362-4881-8393-8efd6e71944d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:27:01.629488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:01.629488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:01.629488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:01.629488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:01.629488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:01.629488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:01.629488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:01.629488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:01.629488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:01.629488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:01.629488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:01.629488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:01.629488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:01.629488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:01.629488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:01.630516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:01.630516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:01.630516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:01.630516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:01.630516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:01.630516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:01.630516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:01.630516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:01.630516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:01.630516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:01.630516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:01.630516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:01.630516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:01.630516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:01.630516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:01.630516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:01.631535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:01.631535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:01.631535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:01.631535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:01.631535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:01.631535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:01.631535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:01.631535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:01.631535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:01.631535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:01.631535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:27:01.631535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:01.631535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 182/369] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:27:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:01.632582 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae24efc-f362-4881-8393-8efd6e71944d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:01.721406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc748d9b-22cf-42f5-bb5b-9d4b7b8bf6c2 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:01.732904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:01.758133 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae24efc-f362-4881-8393-8efd6e71944d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:01.771615 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ac99c7-67d1-4802-9d04-fb46ff79e548 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:01.772230 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ac99c7-67d1-4802-9d04-fb46ff79e548 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:01.772564 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ac99c7-67d1-4802-9d04-fb46ff79e548 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:01.772851 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ac99c7-67d1-4802-9d04-fb46ff79e548 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:01.773580 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ac99c7-67d1-4802-9d04-fb46ff79e548 None admin] RBAC: Adding query filter params (group.id=819dc62c48e64e6a97d7337cb2595c5d, role.id=025f58735650469d9fa6faca3e9e7998, scope.project.id=25655c9cb779428f96b5430e993f1337) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:01.773893 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ac99c7-67d1-4802-9d04-fb46ff79e548 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:01.775391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ac99c7-67d1-4802-9d04-fb46ff79e548 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:01.777870 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:01.782241 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 188/370] 10.222.0.39 () {56 vars in 1376 bytes} [Fri Sep 30 09:27:01 2022] GET /identity/v3/role_assignments?group.id=819dc62c48e64e6a97d7337cb2595c5d&scope.project.id=25655c9cb779428f96b5430e993f1337&role.id=025f58735650469d9fa6faca3e9e7998 => generated 260 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:01.782709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ac99c7-67d1-4802-9d04-fb46ff79e548 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:01.800093 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1cba1f-031c-4fc7-80a4-69aef20d014b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:01.801094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1cba1f-031c-4fc7-80a4-69aef20d014b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:01.801608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1cba1f-031c-4fc7-80a4-69aef20d014b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:01.802028 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1cba1f-031c-4fc7-80a4-69aef20d014b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:01.899318 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1cba1f-031c-4fc7-80a4-69aef20d014b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:01.901968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1cba1f-031c-4fc7-80a4-69aef20d014b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:01.908265 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ac99c7-67d1-4802-9d04-fb46ff79e548 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:01.955283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7058c8-8fcd-4d1d-80d3-b035e436429f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:01.956195 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7058c8-8fcd-4d1d-80d3-b035e436429f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:01.956629 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7058c8-8fcd-4d1d-80d3-b035e436429f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:01.957051 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7058c8-8fcd-4d1d-80d3-b035e436429f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:01.957991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7058c8-8fcd-4d1d-80d3-b035e436429f None admin] RBAC: Adding query filter params (group.id=819dc62c48e64e6a97d7337cb2595c5d, role.id=da6797f7fa24429095283b460be5c792, scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:01.958467 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7058c8-8fcd-4d1d-80d3-b035e436429f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:01.960699 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7058c8-8fcd-4d1d-80d3-b035e436429f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:01.969414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 189/371] 10.222.0.39 () {56 vars in 1376 bytes} [Fri Sep 30 09:27:01 2022] GET /identity/v3/role_assignments?group.id=819dc62c48e64e6a97d7337cb2595c5d&scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211&role.id=da6797f7fa24429095283b460be5c792 => generated 260 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:01.970832 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7058c8-8fcd-4d1d-80d3-b035e436429f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:02.059637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:02.067120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 183/372] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:27:02.076520 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1cba1f-031c-4fc7-80a4-69aef20d014b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:02.100560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:02.172308 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7058c8-8fcd-4d1d-80d3-b035e436429f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:02.223685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1cba1f-031c-4fc7-80a4-69aef20d014b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:02.262406 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c414d09-2ef7-486c-b8f5-371db26e2f19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:02.262638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6057e314-cb29-4445-b58f-fa6098fd6c95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:02.263052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6057e314-cb29-4445-b58f-fa6098fd6c95 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:02.263156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c414d09-2ef7-486c-b8f5-371db26e2f19 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:02.263327 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c414d09-2ef7-486c-b8f5-371db26e2f19 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:02.263443 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6057e314-cb29-4445-b58f-fa6098fd6c95 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:02.263597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c414d09-2ef7-486c-b8f5-371db26e2f19 None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:02.263695 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6057e314-cb29-4445-b58f-fa6098fd6c95 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:02.264357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6057e314-cb29-4445-b58f-fa6098fd6c95 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:02.265499 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6057e314-cb29-4445-b58f-fa6098fd6c95 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:02.265613 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c414d09-2ef7-486c-b8f5-371db26e2f19 None admin] RBAC: Authorizing `identity:get_project(project_id=25655c9cb779428f96b5430e993f1337)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:02.266582 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6057e314-cb29-4445-b58f-fa6098fd6c95 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:02.266714 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c414d09-2ef7-486c-b8f5-371db26e2f19 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:02.268566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 184/373] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:27:02 2022] GET /identity/v3/projects/25655c9cb779428f96b5430e993f1337 => generated 302 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:02.273824 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c414d09-2ef7-486c-b8f5-371db26e2f19 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:02.285919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 190/374] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:27:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:02.291554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6057e314-cb29-4445-b58f-fa6098fd6c95 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:02.367402 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:02.386356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c414d09-2ef7-486c-b8f5-371db26e2f19 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:02.428087 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bbac39-40a4-48d2-bb00-3cd231881028 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:02.428720 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bbac39-40a4-48d2-bb00-3cd231881028 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:02.428996 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bbac39-40a4-48d2-bb00-3cd231881028 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:02.429267 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bbac39-40a4-48d2-bb00-3cd231881028 None admin] PATH_INFO: `/v3/groups/819dc62c48e64e6a97d7337cb2595c5d` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:02.430635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58bbac39-40a4-48d2-bb00-3cd231881028 None admin] RBAC: Authorizing `identity:get_group(group_id=819dc62c48e64e6a97d7337cb2595c5d)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:02.432269 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58bbac39-40a4-48d2-bb00-3cd231881028 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:02.433892 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 185/375] 10.222.0.39 () {56 vars in 1180 bytes} [Fri Sep 30 09:27:02 2022] GET /identity/v3/groups/819dc62c48e64e6a97d7337cb2595c5d => generated 226 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:02.443696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bbac39-40a4-48d2-bb00-3cd231881028 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:02.458618 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:02.464755 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6057e314-cb29-4445-b58f-fa6098fd6c95 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:02.547305 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e32149-9152-4d5c-b4b1-75cc3b23a17b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:02.548314 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e32149-9152-4d5c-b4b1-75cc3b23a17b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:02.548841 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e32149-9152-4d5c-b4b1-75cc3b23a17b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:02.549316 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e32149-9152-4d5c-b4b1-75cc3b23a17b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:02.554572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bbac39-40a4-48d2-bb00-3cd231881028 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:02.611842 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ee0b15-0eb2-4ac7-ab4e-e813cc022d87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:02.612465 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ee0b15-0eb2-4ac7-ab4e-e813cc022d87 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:02.612748 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ee0b15-0eb2-4ac7-ab4e-e813cc022d87 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:02.613018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ee0b15-0eb2-4ac7-ab4e-e813cc022d87 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:02.613739 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ee0b15-0eb2-4ac7-ab4e-e813cc022d87 None admin] RBAC: Adding query filter params (group.id=819dc62c48e64e6a97d7337cb2595c5d, role.id=da6797f7fa24429095283b460be5c792, scope.project.id=25655c9cb779428f96b5430e993f1337) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:02.614048 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ee0b15-0eb2-4ac7-ab4e-e813cc022d87 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:02.615499 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ee0b15-0eb2-4ac7-ab4e-e813cc022d87 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:02.627151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 186/376] 10.222.0.39 () {56 vars in 1376 bytes} [Fri Sep 30 09:27:02 2022] GET /identity/v3/role_assignments?group.id=819dc62c48e64e6a97d7337cb2595c5d&scope.project.id=25655c9cb779428f96b5430e993f1337&role.id=da6797f7fa24429095283b460be5c792 => generated 260 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:02.646498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e32149-9152-4d5c-b4b1-75cc3b23a17b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:02.648264 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e32149-9152-4d5c-b4b1-75cc3b23a17b None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:02.783740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:02.788779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 191/377] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 498 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:27:02.794649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ee0b15-0eb2-4ac7-ab4e-e813cc022d87 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:02.824814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:02.914494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ee0b15-0eb2-4ac7-ab4e-e813cc022d87 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:02.959404 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01efe0b-5269-45a1-a32b-8c441ec5cf72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:02.960042 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01efe0b-5269-45a1-a32b-8c441ec5cf72 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:02.960318 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01efe0b-5269-45a1-a32b-8c441ec5cf72 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:02.960591 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01efe0b-5269-45a1-a32b-8c441ec5cf72 None admin] PATH_INFO: `/v3/projects/f04f50d086c64df3bc18b4c8f8baadc4` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:02.963442 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01efe0b-5269-45a1-a32b-8c441ec5cf72 None admin] RBAC: Authorizing `identity:get_project(project_id=f04f50d086c64df3bc18b4c8f8baadc4)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:02.964885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01efe0b-5269-45a1-a32b-8c441ec5cf72 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:02.966830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 187/378] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:27:02 2022] GET /identity/v3/projects/f04f50d086c64df3bc18b4c8f8baadc4 => generated 344 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:02.973877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e32149-9152-4d5c-b4b1-75cc3b23a17b None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:03.121690 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e32149-9152-4d5c-b4b1-75cc3b23a17b None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:03.180531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e068daa-e27f-4166-a213-58b048a360fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:03.181501 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e068daa-e27f-4166-a213-58b048a360fe None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:03.181975 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e068daa-e27f-4166-a213-58b048a360fe None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:03.182464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e068daa-e27f-4166-a213-58b048a360fe None admin] PATH_INFO: `/v3/groups/327fc8e7db354bc7bd8f500abe10f028` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:03.188824 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e068daa-e27f-4166-a213-58b048a360fe None admin] RBAC: Authorizing `identity:get_group(group_id=327fc8e7db354bc7bd8f500abe10f028)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:03.190672 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e068daa-e27f-4166-a213-58b048a360fe None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:03.192840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 192/379] 10.222.0.39 () {56 vars in 1180 bytes} [Fri Sep 30 09:27:02 2022] GET /identity/v3/groups/327fc8e7db354bc7bd8f500abe10f028 => generated 229 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:03.198395 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01efe0b-5269-45a1-a32b-8c441ec5cf72 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:03.283226 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:03.302998 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01efe0b-5269-45a1-a32b-8c441ec5cf72 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:03.340554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1499b36b-f5e4-4bae-beca-a0450cc525a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:03.341550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1499b36b-f5e4-4bae-beca-a0450cc525a9 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:03.342102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1499b36b-f5e4-4bae-beca-a0450cc525a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:03.342582 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1499b36b-f5e4-4bae-beca-a0450cc525a9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:03.344034 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1499b36b-f5e4-4bae-beca-a0450cc525a9 None admin] RBAC: Adding query filter params (group.id=327fc8e7db354bc7bd8f500abe10f028, role.id=e874c311084c4c9fb7691bce461d319b, scope.project.id=f04f50d086c64df3bc18b4c8f8baadc4) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:03.344549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1499b36b-f5e4-4bae-beca-a0450cc525a9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:03.346233 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1499b36b-f5e4-4bae-beca-a0450cc525a9 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:03.352689 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 188/380] 10.222.0.39 () {56 vars in 1376 bytes} [Fri Sep 30 09:27:03 2022] GET /identity/v3/role_assignments?group.id=327fc8e7db354bc7bd8f500abe10f028&scope.project.id=f04f50d086c64df3bc18b4c8f8baadc4&role.id=e874c311084c4c9fb7691bce461d319b => generated 260 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:03.382645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292388d8-2b40-4c2c-8940-503b2cf14853 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:03.383318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292388d8-2b40-4c2c-8940-503b2cf14853 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:03.383797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292388d8-2b40-4c2c-8940-503b2cf14853 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:03.385403 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 193/381] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:03.391175 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2c6559-2620-4dd2-aca7-3abb4966599f None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:03.391769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2c6559-2620-4dd2-aca7-3abb4966599f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:03.392241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2c6559-2620-4dd2-aca7-3abb4966599f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:03.419790 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-5b2c6559-2620-4dd2-aca7-3abb4966599f None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:03.422023 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5b2c6559-2620-4dd2-aca7-3abb4966599f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:03.512638 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 189/382] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:03.822432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292388d8-2b40-4c2c-8940-503b2cf14853 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:03.910714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:03.957768 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292388d8-2b40-4c2c-8940-503b2cf14853 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:04.005027 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc9757d-64aa-4303-adca-b743ed4bc8ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:04.005670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc9757d-64aa-4303-adca-b743ed4bc8ba None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:04.005935 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc9757d-64aa-4303-adca-b743ed4bc8ba None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:04.006210 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc9757d-64aa-4303-adca-b743ed4bc8ba None admin] PATH_INFO: `/v3/roles/member` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:04.027618 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc9757d-64aa-4303-adca-b743ed4bc8ba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:04.033834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-bdc9757d-64aa-4303-adca-b743ed4bc8ba None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:27:04.033834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:04.033834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:04.033834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:04.033834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:04.033834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:04.033834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:04.033834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:04.033834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:04.033834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:04.033834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:04.033834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:04.033834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:04.033834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:04.033834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:04.033834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:04.034774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:04.034774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:04.034774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:04.034774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:04.034774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:04.034774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:04.034774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:04.034774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:04.034774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:04.034774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:04.034774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:04.034774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:04.034774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:04.034774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:04.034774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:04.034774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:04.035581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:04.035581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:04.035581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:04.035581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:04.035581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:04.035581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:04.035581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:04.035581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:04.035581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:04.035581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:04.035581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:27:04.035581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:04.035581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 194/383] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:27:03 2022] GET /identity/v3/roles/member => generated 84 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:04.041077 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2c6559-2620-4dd2-aca7-3abb4966599f None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:04.130739 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:04.161409 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2c6559-2620-4dd2-aca7-3abb4966599f None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:04.213555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc24a7bb-3506-4cda-b8ec-a4ee7dbb85b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:04.214176 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc24a7bb-3506-4cda-b8ec-a4ee7dbb85b1 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:04.214456 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc24a7bb-3506-4cda-b8ec-a4ee7dbb85b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:04.214768 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc24a7bb-3506-4cda-b8ec-a4ee7dbb85b1 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:04.215434 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc24a7bb-3506-4cda-b8ec-a4ee7dbb85b1 None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:04.215734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc24a7bb-3506-4cda-b8ec-a4ee7dbb85b1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:04.217112 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc24a7bb-3506-4cda-b8ec-a4ee7dbb85b1 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:04.225822 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 190/384] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:27:04 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:04.231338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc9757d-64aa-4303-adca-b743ed4bc8ba None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:04.363179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc9757d-64aa-4303-adca-b743ed4bc8ba None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:04.390940 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:04.402982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5841d59a-9e1c-47af-bda9-f8fdbde7c359 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:04.404836 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5841d59a-9e1c-47af-bda9-f8fdbde7c359 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:04.404836 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5841d59a-9e1c-47af-bda9-f8fdbde7c359 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:04.404836 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5841d59a-9e1c-47af-bda9-f8fdbde7c359 None admin] PATH_INFO: `/v3/groups/819dc62c48e64e6a97d7337cb2595c5d` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:04.405368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5841d59a-9e1c-47af-bda9-f8fdbde7c359 None admin] RBAC: Authorizing `identity:get_group(group_id=819dc62c48e64e6a97d7337cb2595c5d)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:04.406836 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5841d59a-9e1c-47af-bda9-f8fdbde7c359 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:04.408909 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 195/385] 10.222.0.39 () {56 vars in 1180 bytes} [Fri Sep 30 09:27:04 2022] GET /identity/v3/groups/819dc62c48e64e6a97d7337cb2595c5d => generated 226 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:04.414147 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc24a7bb-3506-4cda-b8ec-a4ee7dbb85b1 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:04.566969 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc24a7bb-3506-4cda-b8ec-a4ee7dbb85b1 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:04.593522 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7723c5ca-83f5-4728-8e00-504cccbf1ece None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:04.593899 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7723c5ca-83f5-4728-8e00-504cccbf1ece None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:04.594228 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7723c5ca-83f5-4728-8e00-504cccbf1ece None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:04.595605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 196/386] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:27:04.603979 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24500e8-b910-406f-b56e-ce72961a03b8 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:04.604268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24500e8-b910-406f-b56e-ce72961a03b8 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:04.604542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24500e8-b910-406f-b56e-ce72961a03b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:04.616781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03edd684-46b9-4f01-a066-d71de803672e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:04.617633 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03edd684-46b9-4f01-a066-d71de803672e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:04.617915 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03edd684-46b9-4f01-a066-d71de803672e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:04.618185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03edd684-46b9-4f01-a066-d71de803672e None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:04.620165 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03edd684-46b9-4f01-a066-d71de803672e None admin] RBAC: Authorizing `identity:get_project(project_id=333c93c6a3ef48b6806ea0d5f14ce211)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:04.621593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03edd684-46b9-4f01-a066-d71de803672e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:04.629114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 191/387] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:27:04 2022] GET /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211 => generated 298 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:04.634951 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03edd684-46b9-4f01-a066-d71de803672e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:04.638746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-e24500e8-b910-406f-b56e-ce72961a03b8 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:04.644841 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-e24500e8-b910-406f-b56e-ce72961a03b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:04.645334 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:04.729306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 197/388] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:04.730672 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de479c5-35c6-4f2e-93d0-d5fb1869a185 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:04.730962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de479c5-35c6-4f2e-93d0-d5fb1869a185 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:04.731233 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de479c5-35c6-4f2e-93d0-d5fb1869a185 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:04.732705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 198/389] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:04.739215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b73f71-fcc4-4803-b446-7dd72ab931b5 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:04.739785 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b73f71-fcc4-4803-b446-7dd72ab931b5 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:04.740068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b73f71-fcc4-4803-b446-7dd72ab931b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:04.762789 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03edd684-46b9-4f01-a066-d71de803672e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:04.763887 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-f1b73f71-fcc4-4803-b446-7dd72ab931b5 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:04.765385 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-f1b73f71-fcc4-4803-b446-7dd72ab931b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:04.802655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ca2a0d-96dd-4507-8ddb-505a0e44df0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:04.804090 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ca2a0d-96dd-4507-8ddb-505a0e44df0d None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:04.804616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ca2a0d-96dd-4507-8ddb-505a0e44df0d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:04.805100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ca2a0d-96dd-4507-8ddb-505a0e44df0d None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211/groups/819dc62c48e64e6a97d7337cb2595c5d/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:04.807458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ca2a0d-96dd-4507-8ddb-505a0e44df0d None admin] RBAC: Authorizing `identity:create_grant(project_id=333c93c6a3ef48b6806ea0d5f14ce211, group_id=819dc62c48e64e6a97d7337cb2595c5d, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:04.809261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ca2a0d-96dd-4507-8ddb-505a0e44df0d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:04.853412 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 199/390] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:04.931395 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 192/391] 10.222.0.39 () {58 vars in 1442 bytes} [Fri Sep 30 09:27:04 2022] PUT /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211/groups/819dc62c48e64e6a97d7337cb2595c5d/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:27:05.057770 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b73f71-fcc4-4803-b446-7dd72ab931b5 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:05.127928 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:05.183893 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b73f71-fcc4-4803-b446-7dd72ab931b5 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:05.237900 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d83be3f-0cc4-4c0e-8698-4e95138fb8de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:05.238564 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d83be3f-0cc4-4c0e-8698-4e95138fb8de None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:05.238878 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d83be3f-0cc4-4c0e-8698-4e95138fb8de None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:05.239154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d83be3f-0cc4-4c0e-8698-4e95138fb8de None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:05.252789 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ca2a0d-96dd-4507-8ddb-505a0e44df0d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:05.261906 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d83be3f-0cc4-4c0e-8698-4e95138fb8de None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:05.270108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-9d83be3f-0cc4-4c0e-8698-4e95138fb8de None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:27:05.270108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:05.270108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:05.270108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:05.270108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:05.270108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:05.270108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:05.270108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:05.270108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:05.270108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:05.270108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:05.270108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:05.270108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:05.270108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:05.270108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:05.270108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:05.271063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:05.271063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:05.271063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:05.271063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:05.271063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:05.271063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:05.271063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:05.271063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:05.271063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:05.271063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:05.271063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:05.271063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:05.271063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:05.271063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:05.271063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:05.271063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:05.271882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:05.271882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:05.271882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:05.271882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:05.271882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:05.271882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:05.271882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:05.271882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:05.271882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:05.271882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:05.271882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:27:05.271882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:05.274681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 200/392] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:27:05 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:05.279293 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d83be3f-0cc4-4c0e-8698-4e95138fb8de None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:05.340251 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:05.381013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ca2a0d-96dd-4507-8ddb-505a0e44df0d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:05.421974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d83be3f-0cc4-4c0e-8698-4e95138fb8de None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:05.422358 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fd88f5-12d1-4072-adbf-0883f046b562 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:05.423053 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fd88f5-12d1-4072-adbf-0883f046b562 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:05.423335 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fd88f5-12d1-4072-adbf-0883f046b562 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:05.423608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fd88f5-12d1-4072-adbf-0883f046b562 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:05.438562 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fd88f5-12d1-4072-adbf-0883f046b562 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:05.439827 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:05.446773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-47fd88f5-12d1-4072-adbf-0883f046b562 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:27:05.446773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:05.446773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:05.446773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:05.446773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:05.446773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:05.446773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:05.446773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:05.446773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:05.446773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:05.446773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:05.446773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:05.446773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:05.446773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:05.446773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:05.446773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:05.447677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:05.447677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:05.447677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:05.447677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:05.447677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:05.447677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:05.447677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:05.447677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:05.447677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:05.447677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:05.447677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:05.447677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:05.447677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:05.447677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:05.447677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:05.447677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:05.448564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:05.448564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:05.448564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:05.448564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:05.448564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:05.448564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:05.448564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:05.448564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:05.448564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:05.448564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:05.448564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:27:05.448564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:05.451432 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 193/393] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:27:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:05.454110 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f088ae71-f16e-4d5a-bc71-61b8c3de81b1 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:05.454580 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f088ae71-f16e-4d5a-bc71-61b8c3de81b1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:05.455302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f088ae71-f16e-4d5a-bc71-61b8c3de81b1 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:05.461454 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 194/394] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:05 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:05.461454 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f088ae71-f16e-4d5a-bc71-61b8c3de81b1 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:05.470987 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c836f79-723b-4884-a37a-5bfe44e398f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:05.472223 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c836f79-723b-4884-a37a-5bfe44e398f6 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:05.473359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c836f79-723b-4884-a37a-5bfe44e398f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:05.473876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c836f79-723b-4884-a37a-5bfe44e398f6 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:05.475075 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c836f79-723b-4884-a37a-5bfe44e398f6 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:05.475773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c836f79-723b-4884-a37a-5bfe44e398f6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:05.477572 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c836f79-723b-4884-a37a-5bfe44e398f6 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:05.488915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 201/395] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:05 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:05.493520 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e470611-6972-4a4f-a293-a67bcaa764a6 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:05.493520 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e470611-6972-4a4f-a293-a67bcaa764a6 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:05.493520 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e470611-6972-4a4f-a293-a67bcaa764a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:05.523986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-9e470611-6972-4a4f-a293-a67bcaa764a6 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:05.525191 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-9e470611-6972-4a4f-a293-a67bcaa764a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:05.594916 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f088ae71-f16e-4d5a-bc71-61b8c3de81b1 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:05.602616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 202/396] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:05.602616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e470611-6972-4a4f-a293-a67bcaa764a6 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:05.669488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:05.688362 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328a85ee-b53d-4123-b326-aeb1c197e232 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:05.689251 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328a85ee-b53d-4123-b326-aeb1c197e232 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:05.689486 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328a85ee-b53d-4123-b326-aeb1c197e232 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:05.689760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328a85ee-b53d-4123-b326-aeb1c197e232 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:05.690424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328a85ee-b53d-4123-b326-aeb1c197e232 None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:05.690747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328a85ee-b53d-4123-b326-aeb1c197e232 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:05.695376 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328a85ee-b53d-4123-b326-aeb1c197e232 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:05.704789 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 195/397] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:27:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:05.710355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328a85ee-b53d-4123-b326-aeb1c197e232 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:05.772758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e470611-6972-4a4f-a293-a67bcaa764a6 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:05.778408 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:05.813115 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328a85ee-b53d-4123-b326-aeb1c197e232 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:05.847320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f7bc31-e9cc-4821-a4da-a011fc189825 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:05.847970 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f7bc31-e9cc-4821-a4da-a011fc189825 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:05.848250 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f7bc31-e9cc-4821-a4da-a011fc189825 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:05.848524 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f7bc31-e9cc-4821-a4da-a011fc189825 None admin] PATH_INFO: `/v3/groups/819dc62c48e64e6a97d7337cb2595c5d` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:05.849706 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f7bc31-e9cc-4821-a4da-a011fc189825 None admin] RBAC: Authorizing `identity:get_group(group_id=819dc62c48e64e6a97d7337cb2595c5d)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:05.851194 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f7bc31-e9cc-4821-a4da-a011fc189825 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:05.852759 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 203/398] 10.222.0.39 () {56 vars in 1180 bytes} [Fri Sep 30 09:27:05 2022] GET /identity/v3/groups/819dc62c48e64e6a97d7337cb2595c5d => generated 226 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:05.853691 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc22fda-fe7b-4fe1-972d-b6680ebf9ead None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:05.854621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc22fda-fe7b-4fe1-972d-b6680ebf9ead None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:05.855175 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc22fda-fe7b-4fe1-972d-b6680ebf9ead None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:05.855648 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc22fda-fe7b-4fe1-972d-b6680ebf9ead None admin] PATH_INFO: `/v3/groups/819dc62c48e64e6a97d7337cb2595c5d` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:05.856978 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc22fda-fe7b-4fe1-972d-b6680ebf9ead None admin] RBAC: Authorizing `identity:get_group(group_id=819dc62c48e64e6a97d7337cb2595c5d)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:05.858949 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f7bc31-e9cc-4821-a4da-a011fc189825 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:05.859494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc22fda-fe7b-4fe1-972d-b6680ebf9ead None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:05.861541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 196/399] 10.222.0.39 () {56 vars in 1180 bytes} [Fri Sep 30 09:27:05 2022] GET /identity/v3/groups/819dc62c48e64e6a97d7337cb2595c5d => generated 226 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:05.866979 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc22fda-fe7b-4fe1-972d-b6680ebf9ead None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:05.924933 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:05.972514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc22fda-fe7b-4fe1-972d-b6680ebf9ead None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:06.012519 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f7bc31-e9cc-4821-a4da-a011fc189825 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:06.013573 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bd8ef9-0d47-460d-9ed7-5e7e5f181640 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:06.014240 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bd8ef9-0d47-460d-9ed7-5e7e5f181640 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:06.014562 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bd8ef9-0d47-460d-9ed7-5e7e5f181640 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:06.014958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bd8ef9-0d47-460d-9ed7-5e7e5f181640 None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:06.017920 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bd8ef9-0d47-460d-9ed7-5e7e5f181640 None admin] RBAC: Authorizing `identity:get_project(project_id=25655c9cb779428f96b5430e993f1337)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:06.019768 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bd8ef9-0d47-460d-9ed7-5e7e5f181640 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:06.021551 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 197/400] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:27:05 2022] GET /identity/v3/projects/25655c9cb779428f96b5430e993f1337 => generated 302 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:06.027205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bd8ef9-0d47-460d-9ed7-5e7e5f181640 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:06.074995 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce2fbb3-1593-4437-8be2-52fc261e2384 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:06.075798 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce2fbb3-1593-4437-8be2-52fc261e2384 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:06.076169 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce2fbb3-1593-4437-8be2-52fc261e2384 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:06.077045 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce2fbb3-1593-4437-8be2-52fc261e2384 None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:06.078523 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce2fbb3-1593-4437-8be2-52fc261e2384 None admin] RBAC: Authorizing `identity:get_project(project_id=333c93c6a3ef48b6806ea0d5f14ce211)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:06.080356 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce2fbb3-1593-4437-8be2-52fc261e2384 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:06.082947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 204/401] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:27:05 2022] GET /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211 => generated 298 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:06.088603 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce2fbb3-1593-4437-8be2-52fc261e2384 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:06.136514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:06.145054 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bd8ef9-0d47-460d-9ed7-5e7e5f181640 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:06.169669 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:06.185068 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc7db6b-647e-44e5-a25b-845bcd373385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:06.185758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc7db6b-647e-44e5-a25b-845bcd373385 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:06.186115 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc7db6b-647e-44e5-a25b-845bcd373385 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:06.186980 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc7db6b-647e-44e5-a25b-845bcd373385 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:06.200915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce2fbb3-1593-4437-8be2-52fc261e2384 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:06.202711 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc7db6b-647e-44e5-a25b-845bcd373385 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:06.208067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-3fc7db6b-647e-44e5-a25b-845bcd373385 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:27:06.208067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:06.208067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:06.208067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:06.208067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:06.208067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:06.208067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:06.208067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:06.208067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:06.208067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:06.208067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:06.208067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:06.208067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:06.208067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:06.208067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:06.208067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:06.209136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:06.209136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:06.209136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:06.209136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:06.209136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:06.209136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:06.209136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:06.209136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:06.209136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:06.209136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:06.209136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:06.209136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:06.209136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:06.209136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:06.209136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:06.209136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:06.209955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:06.209955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:06.209955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:06.209955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:06.209955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:06.209955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:06.209955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:06.209955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:06.209955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:06.209955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:06.209955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:27:06.209955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:06.209955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 198/402] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:27:06 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:06.209955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc7db6b-647e-44e5-a25b-845bcd373385 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:06.245444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452ca68e-29f8-4909-b94d-ba26dc428d2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:06.246025 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452ca68e-29f8-4909-b94d-ba26dc428d2c None admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:06.246299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452ca68e-29f8-4909-b94d-ba26dc428d2c None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:06.246742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452ca68e-29f8-4909-b94d-ba26dc428d2c None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337/groups/819dc62c48e64e6a97d7337cb2595c5d/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:06.249378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452ca68e-29f8-4909-b94d-ba26dc428d2c None admin] RBAC: Authorizing `identity:create_grant(project_id=25655c9cb779428f96b5430e993f1337, group_id=819dc62c48e64e6a97d7337cb2595c5d, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:06.250876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452ca68e-29f8-4909-b94d-ba26dc428d2c None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:06.309185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc7db6b-647e-44e5-a25b-845bcd373385 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:06.335854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 205/403] 10.222.0.39 () {58 vars in 1442 bytes} [Fri Sep 30 09:27:06 2022] PUT /identity/v3/projects/25655c9cb779428f96b5430e993f1337/groups/819dc62c48e64e6a97d7337cb2595c5d/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:27:06.336963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ac468a-4bb7-4f09-bd07-c75a3ec94e10 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:06.337314 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ac468a-4bb7-4f09-bd07-c75a3ec94e10 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:06.337662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ac468a-4bb7-4f09-bd07-c75a3ec94e10 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:06.343906 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 206/404] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:06 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:06.344959 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ac468a-4bb7-4f09-bd07-c75a3ec94e10 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:06.347221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d928af71-6911-4840-91c1-7ae99fd53273 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:06.348709 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d928af71-6911-4840-91c1-7ae99fd53273 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:06.350316 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d928af71-6911-4840-91c1-7ae99fd53273 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:06.350821 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d928af71-6911-4840-91c1-7ae99fd53273 None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211/groups/819dc62c48e64e6a97d7337cb2595c5d/roles/da6797f7fa24429095283b460be5c792` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:06.353603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d928af71-6911-4840-91c1-7ae99fd53273 None admin] RBAC: Authorizing `identity:create_grant(project_id=333c93c6a3ef48b6806ea0d5f14ce211, group_id=819dc62c48e64e6a97d7337cb2595c5d, role_id=da6797f7fa24429095283b460be5c792)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:06.356225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d928af71-6911-4840-91c1-7ae99fd53273 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:06.465579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:06.470713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 199/405] 10.222.0.39 () {58 vars in 1442 bytes} [Fri Sep 30 09:27:06 2022] PUT /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211/groups/819dc62c48e64e6a97d7337cb2595c5d/roles/da6797f7fa24429095283b460be5c792 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:27:06.470904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ac468a-4bb7-4f09-bd07-c75a3ec94e10 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:06.472471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2678d66f-4200-4b12-a432-bc4275e021c4 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:06.472907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2678d66f-4200-4b12-a432-bc4275e021c4 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:06.473329 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2678d66f-4200-4b12-a432-bc4275e021c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:06.505532 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-2678d66f-4200-4b12-a432-bc4275e021c4 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:06.509837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2678d66f-4200-4b12-a432-bc4275e021c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:06.537169 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173b3730-d7b6-498b-923b-692a84f1b6b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:06.537800 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173b3730-d7b6-498b-923b-692a84f1b6b7 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:06.538130 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173b3730-d7b6-498b-923b-692a84f1b6b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:06.538411 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173b3730-d7b6-498b-923b-692a84f1b6b7 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:06.540595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173b3730-d7b6-498b-923b-692a84f1b6b7 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:06.540915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173b3730-d7b6-498b-923b-692a84f1b6b7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:06.542353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173b3730-d7b6-498b-923b-692a84f1b6b7 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:06.552124 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 207/406] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:06 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:06.557979 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173b3730-d7b6-498b-923b-692a84f1b6b7 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:06.582372 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:06.623112 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 200/407] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:06.708890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173b3730-d7b6-498b-923b-692a84f1b6b7 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:06.744869 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e983ebba-4da1-41cd-bb22-c1a46bde9786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:06.745728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e983ebba-4da1-41cd-bb22-c1a46bde9786 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:06.746177 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e983ebba-4da1-41cd-bb22-c1a46bde9786 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:06.746618 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e983ebba-4da1-41cd-bb22-c1a46bde9786 None admin] PATH_INFO: `/v3/groups/819dc62c48e64e6a97d7337cb2595c5d` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:06.747889 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e983ebba-4da1-41cd-bb22-c1a46bde9786 None admin] RBAC: Authorizing `identity:get_group(group_id=819dc62c48e64e6a97d7337cb2595c5d)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:06.749574 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e983ebba-4da1-41cd-bb22-c1a46bde9786 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:06.751366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 208/408] 10.222.0.39 () {56 vars in 1180 bytes} [Fri Sep 30 09:27:06 2022] GET /identity/v3/groups/819dc62c48e64e6a97d7337cb2595c5d => generated 226 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:06.757301 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2678d66f-4200-4b12-a432-bc4275e021c4 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:06.896901 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2678d66f-4200-4b12-a432-bc4275e021c4 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:06.941736 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da9b380-bc95-41fd-b3a2-f83069fb7a1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:06.942367 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da9b380-bc95-41fd-b3a2-f83069fb7a1c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:06.942672 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da9b380-bc95-41fd-b3a2-f83069fb7a1c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:06.942945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da9b380-bc95-41fd-b3a2-f83069fb7a1c None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:06.944060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da9b380-bc95-41fd-b3a2-f83069fb7a1c None admin] RBAC: Authorizing `identity:get_project(project_id=25655c9cb779428f96b5430e993f1337)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:06.945477 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da9b380-bc95-41fd-b3a2-f83069fb7a1c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:06.947418 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 201/409] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:27:06 2022] GET /identity/v3/projects/25655c9cb779428f96b5430e993f1337 => generated 302 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:06.953356 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e983ebba-4da1-41cd-bb22-c1a46bde9786 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:06.991292 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da9b380-bc95-41fd-b3a2-f83069fb7a1c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:07.010565 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:07.054954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:07.144955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e983ebba-4da1-41cd-bb22-c1a46bde9786 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:07.159944 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da9b380-bc95-41fd-b3a2-f83069fb7a1c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:07.182616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11521937-13f7-4aaf-acbc-b862d0b795d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:07.182616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11521937-13f7-4aaf-acbc-b862d0b795d4 None admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:07.182616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11521937-13f7-4aaf-acbc-b862d0b795d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:07.182616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11521937-13f7-4aaf-acbc-b862d0b795d4 None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337/groups/819dc62c48e64e6a97d7337cb2595c5d/roles/da6797f7fa24429095283b460be5c792` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:07.184697 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11521937-13f7-4aaf-acbc-b862d0b795d4 None admin] RBAC: Authorizing `identity:create_grant(project_id=25655c9cb779428f96b5430e993f1337, group_id=819dc62c48e64e6a97d7337cb2595c5d, role_id=da6797f7fa24429095283b460be5c792)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:07.186820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11521937-13f7-4aaf-acbc-b862d0b795d4 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:07.216504 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5eef76-ee45-4993-838e-47efef8bfff2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:07.217393 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5eef76-ee45-4993-838e-47efef8bfff2 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:07.217853 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5eef76-ee45-4993-838e-47efef8bfff2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:07.218293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5eef76-ee45-4993-838e-47efef8bfff2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:07.229759 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a5eef76-ee45-4993-838e-47efef8bfff2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:07.235258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-7a5eef76-ee45-4993-838e-47efef8bfff2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:27:07.235258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:07.235258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:07.235258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:07.235258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:07.235258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:07.235258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:07.235258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:07.235258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:07.235258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:07.235258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:07.235258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:07.235258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:07.235258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:07.235258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:07.235258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:07.236447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:07.236447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:07.236447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:07.236447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:07.236447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:07.236447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:07.236447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:07.236447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:07.236447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:07.236447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:07.236447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:07.236447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:07.236447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:07.236447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:07.236447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:07.236447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:07.237496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:07.237496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:07.237496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:07.237496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:07.237496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:07.237496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:07.237496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:07.237496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:07.237496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:07.237496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:07.237496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:27:07.237496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:07.239810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 202/410] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:27:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:07.245662 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5eef76-ee45-4993-838e-47efef8bfff2 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:07.279504 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 209/411] 10.222.0.39 () {58 vars in 1442 bytes} [Fri Sep 30 09:27:06 2022] PUT /identity/v3/projects/25655c9cb779428f96b5430e993f1337/groups/819dc62c48e64e6a97d7337cb2595c5d/roles/da6797f7fa24429095283b460be5c792 => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:27:07.372556 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5eef76-ee45-4993-838e-47efef8bfff2 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:07.375810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:07.413977 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff0eed4-4ae4-488e-ae2b-dffe5c5a40cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:07.414615 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff0eed4-4ae4-488e-ae2b-dffe5c5a40cd None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:07.414910 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff0eed4-4ae4-488e-ae2b-dffe5c5a40cd None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:07.415186 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff0eed4-4ae4-488e-ae2b-dffe5c5a40cd None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:07.415854 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff0eed4-4ae4-488e-ae2b-dffe5c5a40cd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:07.416159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff0eed4-4ae4-488e-ae2b-dffe5c5a40cd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:07.417597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff0eed4-4ae4-488e-ae2b-dffe5c5a40cd None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:07.427844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 203/412] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:27:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:07.433580 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11521937-13f7-4aaf-acbc-b862d0b795d4 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:07.465200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb20681-2d4e-4bac-a4ca-9a89b5233b0c None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:07.465535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb20681-2d4e-4bac-a4ca-9a89b5233b0c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:07.465751 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb20681-2d4e-4bac-a4ca-9a89b5233b0c None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:07.467360 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 204/413] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:07.472509 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b35cdcf-3b09-4911-b5b2-f322c60150b5 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:07.472791 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b35cdcf-3b09-4911-b5b2-f322c60150b5 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:07.473067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b35cdcf-3b09-4911-b5b2-f322c60150b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:07.492286 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-6b35cdcf-3b09-4911-b5b2-f322c60150b5 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:07.493494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6b35cdcf-3b09-4911-b5b2-f322c60150b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:07.564298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:07.576667 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11521937-13f7-4aaf-acbc-b862d0b795d4 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:07.607210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 205/414] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:27:07.621352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729ebc14-e5ba-46a3-a1f4-2676ef11c7dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:07.622041 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729ebc14-e5ba-46a3-a1f4-2676ef11c7dc None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:07.622331 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729ebc14-e5ba-46a3-a1f4-2676ef11c7dc None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:07.622628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729ebc14-e5ba-46a3-a1f4-2676ef11c7dc None admin] PATH_INFO: `/v3/groups/327fc8e7db354bc7bd8f500abe10f028` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:07.624542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729ebc14-e5ba-46a3-a1f4-2676ef11c7dc None admin] RBAC: Authorizing `identity:get_group(group_id=327fc8e7db354bc7bd8f500abe10f028)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:07.626012 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729ebc14-e5ba-46a3-a1f4-2676ef11c7dc None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:07.627994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 210/415] 10.222.0.39 () {56 vars in 1180 bytes} [Fri Sep 30 09:27:07 2022] GET /identity/v3/groups/327fc8e7db354bc7bd8f500abe10f028 => generated 229 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:07.635669 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b35cdcf-3b09-4911-b5b2-f322c60150b5 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:07.738644 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:07.745535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b35cdcf-3b09-4911-b5b2-f322c60150b5 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:07.781087 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8c96b7-62de-4f57-a964-9c1832d980a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:07.787336 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8c96b7-62de-4f57-a964-9c1832d980a0 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:07.787630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8c96b7-62de-4f57-a964-9c1832d980a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:07.787912 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8c96b7-62de-4f57-a964-9c1832d980a0 None admin] PATH_INFO: `/v3/projects/f04f50d086c64df3bc18b4c8f8baadc4` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:07.789125 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8c96b7-62de-4f57-a964-9c1832d980a0 None admin] RBAC: Authorizing `identity:get_project(project_id=f04f50d086c64df3bc18b4c8f8baadc4)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:07.790705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8c96b7-62de-4f57-a964-9c1832d980a0 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:07.792391 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 206/416] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:27:07 2022] GET /identity/v3/projects/f04f50d086c64df3bc18b4c8f8baadc4 => generated 344 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:07.798742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729ebc14-e5ba-46a3-a1f4-2676ef11c7dc None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:07.898046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8c96b7-62de-4f57-a964-9c1832d980a0 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:07.941925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729ebc14-e5ba-46a3-a1f4-2676ef11c7dc None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:07.986566 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ed75a2-5b97-4e9c-a965-e98b8fee5f21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:07.989438 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ed75a2-5b97-4e9c-a965-e98b8fee5f21 None admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:07.991216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ed75a2-5b97-4e9c-a965-e98b8fee5f21 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:07.992367 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ed75a2-5b97-4e9c-a965-e98b8fee5f21 None admin] PATH_INFO: `/v3/projects/f04f50d086c64df3bc18b4c8f8baadc4/groups/327fc8e7db354bc7bd8f500abe10f028/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:07.996855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ed75a2-5b97-4e9c-a965-e98b8fee5f21 None admin] RBAC: Authorizing `identity:create_grant(project_id=f04f50d086c64df3bc18b4c8f8baadc4, group_id=327fc8e7db354bc7bd8f500abe10f028, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:07.999572 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ed75a2-5b97-4e9c-a965-e98b8fee5f21 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:08.021107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8c96b7-62de-4f57-a964-9c1832d980a0 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:08.059219 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda0107f-1cd1-449d-80a8-8da3c3260195 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:08.059899 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda0107f-1cd1-449d-80a8-8da3c3260195 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:08.060177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda0107f-1cd1-449d-80a8-8da3c3260195 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:08.060508 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda0107f-1cd1-449d-80a8-8da3c3260195 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:08.071636 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda0107f-1cd1-449d-80a8-8da3c3260195 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:08.077475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-eda0107f-1cd1-449d-80a8-8da3c3260195 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:27:08.077475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:08.077475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:08.077475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:08.077475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:08.077475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:08.077475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:08.077475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:08.077475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:08.077475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:08.077475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:08.077475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:08.077475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:08.077475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:08.077475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:08.077475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:08.078375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:08.078375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:08.078375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:08.078375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:08.078375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:08.078375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:08.078375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:08.078375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:08.078375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:08.078375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:08.078375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:08.078375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:08.078375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:08.078375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:08.078375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:08.078375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:08.079236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:08.079236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:08.079236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:08.079236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:08.079236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:08.079236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:08.079236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:08.079236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:08.079236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:08.079236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:08.079236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:27:08.079236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:08.079236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 207/417] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:27:07 2022] GET /identity/v3/roles/member => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:08.084935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda0107f-1cd1-449d-80a8-8da3c3260195 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:08.097444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 211/418] 10.222.0.39 () {58 vars in 1442 bytes} [Fri Sep 30 09:27:07 2022] PUT /identity/v3/projects/f04f50d086c64df3bc18b4c8f8baadc4/groups/327fc8e7db354bc7bd8f500abe10f028/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:27:08.136344 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:08.254341 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda0107f-1cd1-449d-80a8-8da3c3260195 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:08.298997 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83675203-7512-44c6-8a58-e74e2749c231 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:08.299612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83675203-7512-44c6-8a58-e74e2749c231 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:08.299945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83675203-7512-44c6-8a58-e74e2749c231 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:08.300235 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83675203-7512-44c6-8a58-e74e2749c231 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:08.300920 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83675203-7512-44c6-8a58-e74e2749c231 None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:08.301229 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83675203-7512-44c6-8a58-e74e2749c231 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:08.302623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83675203-7512-44c6-8a58-e74e2749c231 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:08.313711 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 208/419] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:27:08 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:08.317575 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ed75a2-5b97-4e9c-a965-e98b8fee5f21 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:08.338029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:08.425297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ed75a2-5b97-4e9c-a965-e98b8fee5f21 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:08.461911 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7add94-59dd-49b5-ad35-5e72d6f66426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:08.462439 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7add94-59dd-49b5-ad35-5e72d6f66426 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:08.462753 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7add94-59dd-49b5-ad35-5e72d6f66426 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:08.463029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7add94-59dd-49b5-ad35-5e72d6f66426 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:08.564564 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7add94-59dd-49b5-ad35-5e72d6f66426 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:08.566057 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7add94-59dd-49b5-ad35-5e72d6f66426 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:08.590481 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:08.705855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 212/420] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:27:08.711748 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83675203-7512-44c6-8a58-e74e2749c231 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:08.844850 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:08.866883 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83675203-7512-44c6-8a58-e74e2749c231 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:08.903862 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b6410e-75e8-425e-bf5d-bdacb728d515 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:08.904498 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b6410e-75e8-425e-bf5d-bdacb728d515 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:08.904778 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b6410e-75e8-425e-bf5d-bdacb728d515 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:08.905085 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b6410e-75e8-425e-bf5d-bdacb728d515 None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:08.906215 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b6410e-75e8-425e-bf5d-bdacb728d515 None admin] RBAC: Authorizing `identity:get_project(project_id=333c93c6a3ef48b6806ea0d5f14ce211)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:08.907702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b6410e-75e8-425e-bf5d-bdacb728d515 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:08.909634 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 209/421] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:27:08 2022] GET /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211 => generated 298 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:08.915157 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7add94-59dd-49b5-ad35-5e72d6f66426 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:09.116380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7add94-59dd-49b5-ad35-5e72d6f66426 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:09.141110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:09.164908 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c350ff-d3fb-437b-afb0-b8e945ea72af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:09.165874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c350ff-d3fb-437b-afb0-b8e945ea72af None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:09.166325 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c350ff-d3fb-437b-afb0-b8e945ea72af None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:09.166795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c350ff-d3fb-437b-afb0-b8e945ea72af None admin] PATH_INFO: `/v3/groups/819dc62c48e64e6a97d7337cb2595c5d` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:09.168229 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c350ff-d3fb-437b-afb0-b8e945ea72af None admin] RBAC: Authorizing `identity:get_group(group_id=819dc62c48e64e6a97d7337cb2595c5d)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:09.169879 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c350ff-d3fb-437b-afb0-b8e945ea72af None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:09.172359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 213/422] 10.222.0.39 () {56 vars in 1180 bytes} [Fri Sep 30 09:27:08 2022] GET /identity/v3/groups/819dc62c48e64e6a97d7337cb2595c5d => generated 226 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:09.182634 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b6410e-75e8-425e-bf5d-bdacb728d515 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:09.211839 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c66709c-da7c-401c-89e9-4b4c3a2dc835 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:09.212133 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c66709c-da7c-401c-89e9-4b4c3a2dc835 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:09.212413 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c66709c-da7c-401c-89e9-4b4c3a2dc835 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:09.213884 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 214/423] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:09.219349 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae800f61-490d-4171-a074-cb89f9ce7a92 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:09.219634 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae800f61-490d-4171-a074-cb89f9ce7a92 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:09.219910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae800f61-490d-4171-a074-cb89f9ce7a92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:09.244468 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-ae800f61-490d-4171-a074-cb89f9ce7a92 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:09.245678 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ae800f61-490d-4171-a074-cb89f9ce7a92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:09.332341 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 215/424] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:27:09.338977 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e93153-9bc0-4ac7-a43a-e907d902ec47 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:09.340428 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e93153-9bc0-4ac7-a43a-e907d902ec47 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:09.340870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e93153-9bc0-4ac7-a43a-e907d902ec47 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:09.342988 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 216/425] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:09 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:09.349390 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b90d76c-71a6-4c07-b049-87a3253627d7 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:09.349390 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b90d76c-71a6-4c07-b049-87a3253627d7 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:09.349390 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b90d76c-71a6-4c07-b049-87a3253627d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:09.363343 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b6410e-75e8-425e-bf5d-bdacb728d515 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:09.383250 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-6b90d76c-71a6-4c07-b049-87a3253627d7 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:09.384658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-6b90d76c-71a6-4c07-b049-87a3253627d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:09.426061 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb082f11-21df-4a55-9c19-f6d67155ed41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:09.427792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb082f11-21df-4a55-9c19-f6d67155ed41 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:09.428796 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb082f11-21df-4a55-9c19-f6d67155ed41 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:09.429214 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb082f11-21df-4a55-9c19-f6d67155ed41 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:09.430441 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb082f11-21df-4a55-9c19-f6d67155ed41 None admin] RBAC: Adding query filter params (group.id=819dc62c48e64e6a97d7337cb2595c5d, role.id=025f58735650469d9fa6faca3e9e7998, scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:09.431836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb082f11-21df-4a55-9c19-f6d67155ed41 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:09.433577 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb082f11-21df-4a55-9c19-f6d67155ed41 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:09.451288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 210/426] 10.222.0.39 () {56 vars in 1376 bytes} [Fri Sep 30 09:27:09 2022] GET /identity/v3/role_assignments?group.id=819dc62c48e64e6a97d7337cb2595c5d&scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211&role.id=025f58735650469d9fa6faca3e9e7998 => generated 611 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:09.507201 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 217/427] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:27:09.626794 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb082f11-21df-4a55-9c19-f6d67155ed41 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:09.650455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:09.745287 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb082f11-21df-4a55-9c19-f6d67155ed41 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:09.786150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a2621a-c0e9-490d-a54c-d03026c910d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:09.786819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a2621a-c0e9-490d-a54c-d03026c910d8 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:09.787097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a2621a-c0e9-490d-a54c-d03026c910d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:09.787365 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a2621a-c0e9-490d-a54c-d03026c910d8 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:09.797722 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a2621a-c0e9-490d-a54c-d03026c910d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:09.802629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-15a2621a-c0e9-490d-a54c-d03026c910d8 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:27:09.802629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:09.802629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:09.802629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:09.802629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:09.802629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:09.802629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:09.802629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:09.802629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:09.802629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:09.802629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:09.802629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:09.802629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:09.802629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:09.802629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:09.802629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:09.803611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:09.803611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:09.803611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:09.803611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:09.803611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:09.803611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:09.803611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:09.803611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:09.803611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:09.803611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:09.803611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:09.803611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:09.803611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:09.803611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:09.803611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:09.803611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:09.804426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:09.804426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:09.804426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:09.804426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:09.804426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:09.804426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:09.804426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:09.804426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:09.804426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:09.804426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:09.804426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:27:09.804426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:09.806914 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 211/428] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:27:09 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:09.809799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b90d76c-71a6-4c07-b049-87a3253627d7 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:09.858892 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a2621a-c0e9-490d-a54c-d03026c910d8 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:09.875553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:09.953787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b90d76c-71a6-4c07-b049-87a3253627d7 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:09.956403 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:09.984037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a2621a-c0e9-490d-a54c-d03026c910d8 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:10.003768 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa781c44-0a6d-4046-8efd-00e30383ccec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:10.005184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa781c44-0a6d-4046-8efd-00e30383ccec None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:10.005937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa781c44-0a6d-4046-8efd-00e30383ccec None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:10.006409 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa781c44-0a6d-4046-8efd-00e30383ccec None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:10.007795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa781c44-0a6d-4046-8efd-00e30383ccec None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:10.008691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa781c44-0a6d-4046-8efd-00e30383ccec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:10.010338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa781c44-0a6d-4046-8efd-00e30383ccec None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:10.023970 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 218/429] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:09 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:10.029927 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa781c44-0a6d-4046-8efd-00e30383ccec None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:10.033559 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4993c9ef-7f7b-4bf6-9b58-8efaa3dc5e04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:10.034653 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4993c9ef-7f7b-4bf6-9b58-8efaa3dc5e04 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:10.035419 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4993c9ef-7f7b-4bf6-9b58-8efaa3dc5e04 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:10.035699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4993c9ef-7f7b-4bf6-9b58-8efaa3dc5e04 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:10.051552 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4993c9ef-7f7b-4bf6-9b58-8efaa3dc5e04 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:10.057560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-4993c9ef-7f7b-4bf6-9b58-8efaa3dc5e04 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:27:10.057560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:10.057560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:10.057560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:10.057560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:10.057560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:10.057560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:10.057560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:10.057560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:10.057560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:10.057560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:10.057560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:10.057560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:10.057560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:10.057560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:10.057560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:10.058869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:10.058869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:10.058869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:10.058869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:10.058869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:10.058869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:10.058869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:10.058869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:10.058869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:10.058869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:10.058869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:10.058869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:10.058869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:10.058869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:10.058869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:10.058869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:10.059729 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:10.059729 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:10.059729 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:10.059729 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:10.059729 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:10.059729 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:10.059729 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:10.059729 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:10.059729 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:10.059729 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:10.059729 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 30 09:27:10.059729 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:10.059729 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 212/430] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:27:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:10.063669 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4993c9ef-7f7b-4bf6-9b58-8efaa3dc5e04 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:10.184733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa781c44-0a6d-4046-8efd-00e30383ccec None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:10.199029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:10.205107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4993c9ef-7f7b-4bf6-9b58-8efaa3dc5e04 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:10.235330 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2ba534-b8c1-4210-960f-0a61e10149a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:10.236306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2ba534-b8c1-4210-960f-0a61e10149a6 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:10.237067 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2ba534-b8c1-4210-960f-0a61e10149a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:10.237563 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2ba534-b8c1-4210-960f-0a61e10149a6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:10.253814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a4a658-2de2-4a01-999f-8ff4a7c07b8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:10.254762 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a4a658-2de2-4a01-999f-8ff4a7c07b8e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:10.255047 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a4a658-2de2-4a01-999f-8ff4a7c07b8e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:10.255378 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a4a658-2de2-4a01-999f-8ff4a7c07b8e None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:10.259718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a4a658-2de2-4a01-999f-8ff4a7c07b8e None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:10.260046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a4a658-2de2-4a01-999f-8ff4a7c07b8e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:10.261659 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a4a658-2de2-4a01-999f-8ff4a7c07b8e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:10.268534 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:10.279238 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 213/431] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:27:10 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:10.279238 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d048b5-d2cc-4037-9aae-cf82cb586534 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:10.279238 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d048b5-d2cc-4037-9aae-cf82cb586534 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:10.279238 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d048b5-d2cc-4037-9aae-cf82cb586534 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:10.283287 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 214/432] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:10 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:10.283603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d048b5-d2cc-4037-9aae-cf82cb586534 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:10.345730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2ba534-b8c1-4210-960f-0a61e10149a6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:10.347272 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2ba534-b8c1-4210-960f-0a61e10149a6 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:10.411223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d048b5-d2cc-4037-9aae-cf82cb586534 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:10.448108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5481391-b144-4a82-ad76-d28b63b7613a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:10.448809 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5481391-b144-4a82-ad76-d28b63b7613a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:10.448989 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5481391-b144-4a82-ad76-d28b63b7613a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:10.449264 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5481391-b144-4a82-ad76-d28b63b7613a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:10.455589 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 219/433] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:27:10.456630 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a05a39-be29-4561-b792-35218bd7dc40 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:10.456925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a05a39-be29-4561-b792-35218bd7dc40 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:10.457205 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a05a39-be29-4561-b792-35218bd7dc40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:10.475217 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-59a05a39-be29-4561-b792-35218bd7dc40 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:10.476408 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-59a05a39-be29-4561-b792-35218bd7dc40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:10.478484 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:10.525143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:10.528737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5481391-b144-4a82-ad76-d28b63b7613a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:10.530258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5481391-b144-4a82-ad76-d28b63b7613a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:10.562128 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 220/434] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:10.569070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a05a39-be29-4561-b792-35218bd7dc40 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:10.658062 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 215/435] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:27:10.660843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5481391-b144-4a82-ad76-d28b63b7613a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:10.694457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a05a39-be29-4561-b792-35218bd7dc40 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:10.730119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb039b92-0887-40ac-b472-b49a6967b211 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:10.730986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb039b92-0887-40ac-b472-b49a6967b211 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:10.731425 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb039b92-0887-40ac-b472-b49a6967b211 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:10.731845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb039b92-0887-40ac-b472-b49a6967b211 None admin] PATH_INFO: `/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:10.733741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb039b92-0887-40ac-b472-b49a6967b211 None admin] RBAC: Authorizing `identity:get_project(project_id=333c93c6a3ef48b6806ea0d5f14ce211)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:10.735364 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb039b92-0887-40ac-b472-b49a6967b211 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:10.737114 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 221/436] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:27:10 2022] GET /identity/v3/projects/333c93c6a3ef48b6806ea0d5f14ce211 => generated 298 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:10.738351 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddfe25e-6797-4db5-8d80-7f642a984f86 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:10.738647 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddfe25e-6797-4db5-8d80-7f642a984f86 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:10.738920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddfe25e-6797-4db5-8d80-7f642a984f86 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:10.740237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 222/437] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:10 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:10.742485 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddfe25e-6797-4db5-8d80-7f642a984f86 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:10.754843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:10.771831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:10.775740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5481391-b144-4a82-ad76-d28b63b7613a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:10.810916 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9c6ae2-95f1-4d6d-a004-8b4b89dbed70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:10.811561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9c6ae2-95f1-4d6d-a004-8b4b89dbed70 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:10.811837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9c6ae2-95f1-4d6d-a004-8b4b89dbed70 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:10.812111 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9c6ae2-95f1-4d6d-a004-8b4b89dbed70 None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:10.813201 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9c6ae2-95f1-4d6d-a004-8b4b89dbed70 None admin] RBAC: Authorizing `identity:get_project(project_id=25655c9cb779428f96b5430e993f1337)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:10.814705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9c6ae2-95f1-4d6d-a004-8b4b89dbed70 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:10.820741 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 216/438] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:27:10 2022] GET /identity/v3/projects/25655c9cb779428f96b5430e993f1337 => generated 302 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:10.821900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faca7f37-c1a8-4bb0-9790-db24aa744d59 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:10.822181 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faca7f37-c1a8-4bb0-9790-db24aa744d59 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:10.822452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faca7f37-c1a8-4bb0-9790-db24aa744d59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:10.839897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-faca7f37-c1a8-4bb0-9790-db24aa744d59 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:10.841090 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-faca7f37-c1a8-4bb0-9790-db24aa744d59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:10.853016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddfe25e-6797-4db5-8d80-7f642a984f86 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:10.886821 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddac663-e6ef-42f0-8983-0b4117b2e14c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:10.887444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddac663-e6ef-42f0-8983-0b4117b2e14c None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:10.887723 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddac663-e6ef-42f0-8983-0b4117b2e14c None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:10.887997 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddac663-e6ef-42f0-8983-0b4117b2e14c None admin] PATH_INFO: `/v3/groups/819dc62c48e64e6a97d7337cb2595c5d` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:10.889102 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddac663-e6ef-42f0-8983-0b4117b2e14c None admin] RBAC: Authorizing `identity:get_group(group_id=819dc62c48e64e6a97d7337cb2595c5d)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:10.890543 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddac663-e6ef-42f0-8983-0b4117b2e14c None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:10.892112 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 223/439] 10.222.0.39 () {56 vars in 1180 bytes} [Fri Sep 30 09:27:10 2022] GET /identity/v3/groups/819dc62c48e64e6a97d7337cb2595c5d => generated 226 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:10.892862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddac663-e6ef-42f0-8983-0b4117b2e14c None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:10.910979 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 217/440] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:10.911801 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faca7f37-c1a8-4bb0-9790-db24aa744d59 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:10.979856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:11.005575 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddac663-e6ef-42f0-8983-0b4117b2e14c None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:11.010382 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:11.015982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faca7f37-c1a8-4bb0-9790-db24aa744d59 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:11.042061 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2247933-89b0-452e-8613-3139088ec9fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:11.042718 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2247933-89b0-452e-8613-3139088ec9fc None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:11.043002 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2247933-89b0-452e-8613-3139088ec9fc None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:11.043333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2247933-89b0-452e-8613-3139088ec9fc None admin] PATH_INFO: `/v3/groups/819dc62c48e64e6a97d7337cb2595c5d` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:11.044443 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2247933-89b0-452e-8613-3139088ec9fc None admin] RBAC: Authorizing `identity:get_group(group_id=819dc62c48e64e6a97d7337cb2595c5d)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:11.046451 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2247933-89b0-452e-8613-3139088ec9fc None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:11.048223 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 224/441] 10.222.0.39 () {56 vars in 1180 bytes} [Fri Sep 30 09:27:10 2022] GET /identity/v3/groups/819dc62c48e64e6a97d7337cb2595c5d => generated 226 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:11.048793 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2247933-89b0-452e-8613-3139088ec9fc None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:11.056898 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcec2595-5234-4184-9188-e353a574870c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:11.057544 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcec2595-5234-4184-9188-e353a574870c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:11.057879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcec2595-5234-4184-9188-e353a574870c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:11.058163 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcec2595-5234-4184-9188-e353a574870c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:11.068449 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcec2595-5234-4184-9188-e353a574870c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:11.073248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-fcec2595-5234-4184-9188-e353a574870c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:27:11.073248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:11.073248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:11.073248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:11.073248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:11.073248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:11.073248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:11.073248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:11.073248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:11.073248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:11.073248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:11.073248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:11.073248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:11.073248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:11.073248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:11.073248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:11.074256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:11.074256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:11.074256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:11.074256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:11.074256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:11.074256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:11.074256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:11.074256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:11.074256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:11.074256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:11.074256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:11.074256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:11.074256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:11.074256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:11.074256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:11.074256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:11.075102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:11.075102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:11.075102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:11.075102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:11.075102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:11.075102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:11.075102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:11.075102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:11.075102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:11.075102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:11.075102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:27:11.075102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:11.075102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 218/442] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:27:10 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:11.075102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcec2595-5234-4184-9188-e353a574870c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:11.161370 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2247933-89b0-452e-8613-3139088ec9fc None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:11.184797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcec2595-5234-4184-9188-e353a574870c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:11.202977 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef10b8a-578e-4f1a-be83-39bf446b20f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:11.203831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef10b8a-578e-4f1a-be83-39bf446b20f9 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:11.204221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef10b8a-578e-4f1a-be83-39bf446b20f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:11.204598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef10b8a-578e-4f1a-be83-39bf446b20f9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:11.205506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef10b8a-578e-4f1a-be83-39bf446b20f9 None admin] RBAC: Adding query filter params (group.id=819dc62c48e64e6a97d7337cb2595c5d, role.id=da6797f7fa24429095283b460be5c792, scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:11.205817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef10b8a-578e-4f1a-be83-39bf446b20f9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:11.207294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef10b8a-578e-4f1a-be83-39bf446b20f9 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:11.213060 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 225/443] 10.222.0.39 () {56 vars in 1376 bytes} [Fri Sep 30 09:27:11 2022] GET /identity/v3/role_assignments?group.id=819dc62c48e64e6a97d7337cb2595c5d&scope.project.id=333c93c6a3ef48b6806ea0d5f14ce211&role.id=da6797f7fa24429095283b460be5c792 => generated 611 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:11.213822 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef10b8a-578e-4f1a-be83-39bf446b20f9 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:11.220936 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01046ceb-0e96-432c-899c-ad086e1f6731 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:11.221573 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01046ceb-0e96-432c-899c-ad086e1f6731 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:11.221854 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01046ceb-0e96-432c-899c-ad086e1f6731 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:11.222129 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01046ceb-0e96-432c-899c-ad086e1f6731 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:11.222907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01046ceb-0e96-432c-899c-ad086e1f6731 None admin] RBAC: Adding query filter params (group.id=819dc62c48e64e6a97d7337cb2595c5d, role.id=025f58735650469d9fa6faca3e9e7998, scope.project.id=25655c9cb779428f96b5430e993f1337) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:11.223222 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01046ceb-0e96-432c-899c-ad086e1f6731 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:11.224616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:11.224744 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01046ceb-0e96-432c-899c-ad086e1f6731 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:11.231525 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 219/444] 10.222.0.39 () {56 vars in 1376 bytes} [Fri Sep 30 09:27:11 2022] GET /identity/v3/role_assignments?group.id=819dc62c48e64e6a97d7337cb2595c5d&scope.project.id=25655c9cb779428f96b5430e993f1337&role.id=025f58735650469d9fa6faca3e9e7998 => generated 611 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:11.232280 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01046ceb-0e96-432c-899c-ad086e1f6731 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:11.260187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:11.353413 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01046ceb-0e96-432c-899c-ad086e1f6731 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:11.365954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef10b8a-578e-4f1a-be83-39bf446b20f9 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:11.393739 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4acedb-8ebf-475f-aa10-51d448a4e52b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:11.394391 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4acedb-8ebf-475f-aa10-51d448a4e52b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:11.394710 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4acedb-8ebf-475f-aa10-51d448a4e52b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:11.394988 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4acedb-8ebf-475f-aa10-51d448a4e52b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:11.410420 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4acedb-8ebf-475f-aa10-51d448a4e52b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:11.412597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-bc4acedb-8ebf-475f-aa10-51d448a4e52b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:27:11.412597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:11.412597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:11.412597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:11.412597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:11.412597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:11.412597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:11.412597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:11.412597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:11.412597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:11.412597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:11.412597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:11.412597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:11.412597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:11.412597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:11.412597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:11.413708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:11.413708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:11.413708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:11.413708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:11.413708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:11.413708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:11.413708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:11.413708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:11.413708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:11.413708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:11.413708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:11.413708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:11.413708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:11.413708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:11.413708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:11.413708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:11.415044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:11.415044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:11.415044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:11.415044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:11.415044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:11.415044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:11.415044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:11.415044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:11.415044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:11.415044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:11.415044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:27:11.415044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:11.415044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 220/445] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:27:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:11.418354 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395f8e2c-7aa3-467d-ad8f-20cef3bfe6a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:11.419129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395f8e2c-7aa3-467d-ad8f-20cef3bfe6a0 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:11.419415 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395f8e2c-7aa3-467d-ad8f-20cef3bfe6a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:11.419689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395f8e2c-7aa3-467d-ad8f-20cef3bfe6a0 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:11.419786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4acedb-8ebf-475f-aa10-51d448a4e52b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:11.420408 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395f8e2c-7aa3-467d-ad8f-20cef3bfe6a0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:11.420770 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395f8e2c-7aa3-467d-ad8f-20cef3bfe6a0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:11.422248 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395f8e2c-7aa3-467d-ad8f-20cef3bfe6a0 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:11.435141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 226/446] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:11 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:11.441608 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395f8e2c-7aa3-467d-ad8f-20cef3bfe6a0 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:11.525878 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:11.530640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:11.552302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4acedb-8ebf-475f-aa10-51d448a4e52b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:11.563964 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395f8e2c-7aa3-467d-ad8f-20cef3bfe6a0 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:11.589907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3c0586-67dc-4fc1-9133-17ee15dd32ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:11.590521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3c0586-67dc-4fc1-9133-17ee15dd32ad None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:11.590852 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3c0586-67dc-4fc1-9133-17ee15dd32ad None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:11.591450 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3c0586-67dc-4fc1-9133-17ee15dd32ad None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:11.592450 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3c0586-67dc-4fc1-9133-17ee15dd32ad None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:11.593151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3c0586-67dc-4fc1-9133-17ee15dd32ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:11.594698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3c0586-67dc-4fc1-9133-17ee15dd32ad None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:11.603166 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d651a3-c257-4cc0-ac52-9b8067d95261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:11.604958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 221/447] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:27:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:11.605084 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d651a3-c257-4cc0-ac52-9b8067d95261 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:11.605084 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d651a3-c257-4cc0-ac52-9b8067d95261 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:11.605084 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d651a3-c257-4cc0-ac52-9b8067d95261 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:11.609832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3c0586-67dc-4fc1-9133-17ee15dd32ad None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:11.688648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d651a3-c257-4cc0-ac52-9b8067d95261 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:11.690667 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d651a3-c257-4cc0-ac52-9b8067d95261 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:11.737957 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3c0586-67dc-4fc1-9133-17ee15dd32ad None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:11.767929 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:11.780335 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f88d454-ff2c-42f5-9a5d-d19c06152e90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:11.780880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f88d454-ff2c-42f5-9a5d-d19c06152e90 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:11.781151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f88d454-ff2c-42f5-9a5d-d19c06152e90 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:11.781427 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f88d454-ff2c-42f5-9a5d-d19c06152e90 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:11.795504 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:11.807863 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 227/448] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 30 09:27:11.813955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d651a3-c257-4cc0-ac52-9b8067d95261 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:11.851054 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f88d454-ff2c-42f5-9a5d-d19c06152e90 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:11.852509 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f88d454-ff2c-42f5-9a5d-d19c06152e90 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:11.909483 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d651a3-c257-4cc0-ac52-9b8067d95261 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:11.943572 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20594e69-0155-47a9-a66c-0835a17fb898 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:11.944190 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20594e69-0155-47a9-a66c-0835a17fb898 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:11.944537 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20594e69-0155-47a9-a66c-0835a17fb898 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:11.944820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20594e69-0155-47a9-a66c-0835a17fb898 None admin] PATH_INFO: `/v3/projects/25655c9cb779428f96b5430e993f1337` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:11.945976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20594e69-0155-47a9-a66c-0835a17fb898 None admin] RBAC: Authorizing `identity:get_project(project_id=25655c9cb779428f96b5430e993f1337)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:11.947486 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20594e69-0155-47a9-a66c-0835a17fb898 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:11.948283 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 222/449] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 30 09:27:11.949163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 228/450] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:27:11 2022] GET /identity/v3/projects/25655c9cb779428f96b5430e993f1337 => generated 302 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:11.953618 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f88d454-ff2c-42f5-9a5d-d19c06152e90 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:11.954078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20594e69-0155-47a9-a66c-0835a17fb898 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:11.987743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:11.994444 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:12.049853 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f88d454-ff2c-42f5-9a5d-d19c06152e90 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:12.050635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20594e69-0155-47a9-a66c-0835a17fb898 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:12.083452 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49101659-4394-4558-bf0c-a4dbdce84c1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:12.084073 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49101659-4394-4558-bf0c-a4dbdce84c1e None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:12.084258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5859152-bc2b-4fb2-b1e1-db51428f0ff2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:12.084392 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49101659-4394-4558-bf0c-a4dbdce84c1e None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:12.084620 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49101659-4394-4558-bf0c-a4dbdce84c1e None admin] PATH_INFO: `/v3/groups/819dc62c48e64e6a97d7337cb2595c5d` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:12.084871 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5859152-bc2b-4fb2-b1e1-db51428f0ff2 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:12.085146 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5859152-bc2b-4fb2-b1e1-db51428f0ff2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:12.085415 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5859152-bc2b-4fb2-b1e1-db51428f0ff2 None admin] PATH_INFO: `/v3/projects/f04f50d086c64df3bc18b4c8f8baadc4` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:12.085723 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49101659-4394-4558-bf0c-a4dbdce84c1e None admin] RBAC: Authorizing `identity:get_group(group_id=819dc62c48e64e6a97d7337cb2595c5d)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:12.086505 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5859152-bc2b-4fb2-b1e1-db51428f0ff2 None admin] RBAC: Authorizing `identity:get_project(project_id=f04f50d086c64df3bc18b4c8f8baadc4)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:12.087217 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49101659-4394-4558-bf0c-a4dbdce84c1e None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:12.087958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5859152-bc2b-4fb2-b1e1-db51428f0ff2 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:12.088979 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 229/451] 10.222.0.39 () {56 vars in 1180 bytes} [Fri Sep 30 09:27:11 2022] GET /identity/v3/groups/819dc62c48e64e6a97d7337cb2595c5d => generated 226 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:12.090617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 223/452] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:27:11 2022] GET /identity/v3/projects/f04f50d086c64df3bc18b4c8f8baadc4 => generated 344 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:12.094528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49101659-4394-4558-bf0c-a4dbdce84c1e None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:12.094672 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5859152-bc2b-4fb2-b1e1-db51428f0ff2 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:12.191124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5859152-bc2b-4fb2-b1e1-db51428f0ff2 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:12.194265 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49101659-4394-4558-bf0c-a4dbdce84c1e None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:12.197621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:12.201736 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:12.224616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ff3f3f-222e-4d63-af1e-76f8b4e53f40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:12.225283 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ff3f3f-222e-4d63-af1e-76f8b4e53f40 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:12.225556 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ff3f3f-222e-4d63-af1e-76f8b4e53f40 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:12.225829 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ff3f3f-222e-4d63-af1e-76f8b4e53f40 None admin] PATH_INFO: `/v3/groups/327fc8e7db354bc7bd8f500abe10f028` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:12.226950 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ff3f3f-222e-4d63-af1e-76f8b4e53f40 None admin] RBAC: Authorizing `identity:get_group(group_id=327fc8e7db354bc7bd8f500abe10f028)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:12.228568 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ff3f3f-222e-4d63-af1e-76f8b4e53f40 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:12.228981 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf01d135-33ba-4f3c-ae45-53016718eb17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:12.229578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf01d135-33ba-4f3c-ae45-53016718eb17 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:12.229858 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf01d135-33ba-4f3c-ae45-53016718eb17 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:12.230117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 224/453] 10.222.0.39 () {56 vars in 1180 bytes} [Fri Sep 30 09:27:12 2022] GET /identity/v3/groups/327fc8e7db354bc7bd8f500abe10f028 => generated 229 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:12.230203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf01d135-33ba-4f3c-ae45-53016718eb17 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:12.230869 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf01d135-33ba-4f3c-ae45-53016718eb17 None admin] RBAC: Adding query filter params (group.id=819dc62c48e64e6a97d7337cb2595c5d, role.id=da6797f7fa24429095283b460be5c792, scope.project.id=25655c9cb779428f96b5430e993f1337) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:12.231176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf01d135-33ba-4f3c-ae45-53016718eb17 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:12.232632 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf01d135-33ba-4f3c-ae45-53016718eb17 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:12.236578 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ff3f3f-222e-4d63-af1e-76f8b4e53f40 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:12.250621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 230/454] 10.222.0.39 () {56 vars in 1376 bytes} [Fri Sep 30 09:27:12 2022] GET /identity/v3/role_assignments?group.id=819dc62c48e64e6a97d7337cb2595c5d&scope.project.id=25655c9cb779428f96b5430e993f1337&role.id=da6797f7fa24429095283b460be5c792 => generated 611 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:12.339667 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ff3f3f-222e-4d63-af1e-76f8b4e53f40 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:12.374868 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c432c543-b2fe-4372-a217-88cc264cc0ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:12.375484 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c432c543-b2fe-4372-a217-88cc264cc0ca None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:12.375761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c432c543-b2fe-4372-a217-88cc264cc0ca None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:12.376029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c432c543-b2fe-4372-a217-88cc264cc0ca None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:12.376756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c432c543-b2fe-4372-a217-88cc264cc0ca None admin] RBAC: Adding query filter params (group.id=327fc8e7db354bc7bd8f500abe10f028, role.id=e874c311084c4c9fb7691bce461d319b, scope.project.id=f04f50d086c64df3bc18b4c8f8baadc4) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:12.377059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c432c543-b2fe-4372-a217-88cc264cc0ca None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:12.378526 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c432c543-b2fe-4372-a217-88cc264cc0ca None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:12.385371 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 225/455] 10.222.0.39 () {56 vars in 1376 bytes} [Fri Sep 30 09:27:12 2022] GET /identity/v3/role_assignments?group.id=327fc8e7db354bc7bd8f500abe10f028&scope.project.id=f04f50d086c64df3bc18b4c8f8baadc4&role.id=e874c311084c4c9fb7691bce461d319b => generated 611 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:14.955069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c9543c-4104-4085-99f3-cd2bead19ea4 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:14.956111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c9543c-4104-4085-99f3-cd2bead19ea4 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:14.956601 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c9543c-4104-4085-99f3-cd2bead19ea4 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:14.958442 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 231/456] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:14.963628 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903bc026-85b9-422e-8deb-2872cbbf1f62 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:14.964448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903bc026-85b9-422e-8deb-2872cbbf1f62 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:14.964939 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903bc026-85b9-422e-8deb-2872cbbf1f62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:14.987142 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-903bc026-85b9-422e-8deb-2872cbbf1f62 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:14.988659 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-903bc026-85b9-422e-8deb-2872cbbf1f62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:15.039779 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:15.083678 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 226/457] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:15.234631 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268e162e-a43c-4d28-aea6-e4a6d7ec53ca None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:15.235246 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268e162e-a43c-4d28-aea6-e4a6d7ec53ca None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:15.235744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268e162e-a43c-4d28-aea6-e4a6d7ec53ca None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:15.237554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 232/458] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:15.242562 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c98b986-1396-4122-876a-53a550b3be12 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:15.243087 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c98b986-1396-4122-876a-53a550b3be12 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:15.243561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c98b986-1396-4122-876a-53a550b3be12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:15.267261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-5c98b986-1396-4122-876a-53a550b3be12 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:15.268779 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5c98b986-1396-4122-876a-53a550b3be12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:15.372721 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 227/459] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:15.526810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268e162e-a43c-4d28-aea6-e4a6d7ec53ca None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:15.538996 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf457d1-faf6-4d43-9454-8540c7d709dc None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:15.539641 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf457d1-faf6-4d43-9454-8540c7d709dc None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:15.540141 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf457d1-faf6-4d43-9454-8540c7d709dc None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:15.542032 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 228/460] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:15.548610 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04af4d3-cd5e-45b5-87ad-5ec23dba36f2 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:15.549443 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04af4d3-cd5e-45b5-87ad-5ec23dba36f2 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:15.549927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04af4d3-cd5e-45b5-87ad-5ec23dba36f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:15.575315 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-a04af4d3-cd5e-45b5-87ad-5ec23dba36f2 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:15.579212 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a04af4d3-cd5e-45b5-87ad-5ec23dba36f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:15.688492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268e162e-a43c-4d28-aea6-e4a6d7ec53ca None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:15.689430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 229/461] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:27:15.706880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04af4d3-cd5e-45b5-87ad-5ec23dba36f2 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:15.737736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0f14aa-5a75-4c81-a7f9-e5a325e166bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:15.738373 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0f14aa-5a75-4c81-a7f9-e5a325e166bb None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:15.738687 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0f14aa-5a75-4c81-a7f9-e5a325e166bb None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:15.738963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0f14aa-5a75-4c81-a7f9-e5a325e166bb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:15.750753 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0f14aa-5a75-4c81-a7f9-e5a325e166bb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:15.752541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0f14aa-5a75-4c81-a7f9-e5a325e166bb None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:15.769650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-ae0f14aa-5a75-4c81-a7f9-e5a325e166bb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:15.769650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:15.769650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:15.769650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:15.769650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:15.769650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:15.769650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:15.769650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:15.769650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:15.769650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:15.769650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:15.769650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:15.769650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:15.769650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:15.769650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:15.769650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:15.770605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:15.770605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:15.770605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:15.770605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:15.770605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:15.770605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:15.770605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:15.770605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:15.770605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:15.770605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:15.770605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:15.770605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:15.770605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:15.770605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:15.770605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:15.770605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:15.770605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:15.771506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:15.771506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:15.771506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:15.771932 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:15.773730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 233/462] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:15.787165 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0f14aa-5a75-4c81-a7f9-e5a325e166bb None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:15.825594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:15.895999 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04af4d3-cd5e-45b5-87ad-5ec23dba36f2 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:15.953693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0f14aa-5a75-4c81-a7f9-e5a325e166bb None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:15.959513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b073f0-00a7-4996-864b-44512188ac21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:15.961563 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b073f0-00a7-4996-864b-44512188ac21 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:15.966150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b073f0-00a7-4996-864b-44512188ac21 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:15.967663 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b073f0-00a7-4996-864b-44512188ac21 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:15.987095 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b073f0-00a7-4996-864b-44512188ac21 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:15.988924 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b073f0-00a7-4996-864b-44512188ac21 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:16.015178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-f1b073f0-00a7-4996-864b-44512188ac21 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:16.015178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:16.015178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:16.015178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:16.015178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:16.015178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:16.015178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:16.015178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:16.015178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:16.015178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:16.015178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:16.015178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:16.015178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:16.015178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:16.015178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:16.015178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:16.016156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:16.016156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:16.016156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:16.016156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:16.016156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:16.016156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:16.016156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:16.016156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:16.016156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:16.016156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:16.016156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:16.016156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:16.016156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:16.016156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:16.016156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:16.016156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:16.016156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:16.017018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:16.017018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:16.017018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:16.017181 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc9cd73-6b4d-47df-bfa9-cc18c01cc820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:16.017181 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc9cd73-6b4d-47df-bfa9-cc18c01cc820 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:16.017181 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc9cd73-6b4d-47df-bfa9-cc18c01cc820 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:16.017181 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc9cd73-6b4d-47df-bfa9-cc18c01cc820 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:16.019461 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc9cd73-6b4d-47df-bfa9-cc18c01cc820 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:16.019655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 230/463] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 313 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:16.020103 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc9cd73-6b4d-47df-bfa9-cc18c01cc820 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:16.020563 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b073f0-00a7-4996-864b-44512188ac21 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:16.021735 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc9cd73-6b4d-47df-bfa9-cc18c01cc820 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:16.041064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 234/464] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:16.042324 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc9cd73-6b4d-47df-bfa9-cc18c01cc820 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:16.134262 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:16.152301 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:16.156527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc9cd73-6b4d-47df-bfa9-cc18c01cc820 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:16.171020 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b073f0-00a7-4996-864b-44512188ac21 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:16.203321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98cf11aa-0bc7-4e1d-bd64-ef0613121836 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:16.204688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cf11aa-0bc7-4e1d-bd64-ef0613121836 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:16.205184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cf11aa-0bc7-4e1d-bd64-ef0613121836 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:16.205896 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cf11aa-0bc7-4e1d-bd64-ef0613121836 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:16.207010 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98cf11aa-0bc7-4e1d-bd64-ef0613121836 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:16.207511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98cf11aa-0bc7-4e1d-bd64-ef0613121836 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:16.209735 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98cf11aa-0bc7-4e1d-bd64-ef0613121836 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:16.219396 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a24abb8-2f7f-4caf-bea7-e0777bd4f7db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:16.220245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a24abb8-2f7f-4caf-bea7-e0777bd4f7db None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:16.220734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a24abb8-2f7f-4caf-bea7-e0777bd4f7db None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:16.221191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a24abb8-2f7f-4caf-bea7-e0777bd4f7db None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:16.230533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 235/465] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:16.232191 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98cf11aa-0bc7-4e1d-bd64-ef0613121836 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:16.235679 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a24abb8-2f7f-4caf-bea7-e0777bd4f7db None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:16.238522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a24abb8-2f7f-4caf-bea7-e0777bd4f7db None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:16.264074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-5a24abb8-2f7f-4caf-bea7-e0777bd4f7db None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:16.264074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:16.264074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:16.264074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:16.264074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:16.264074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:16.264074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:16.264074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:16.264074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:16.264074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:16.264074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:16.264074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:16.264074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:16.264074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:16.264074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:16.264074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:16.265198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:16.265198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:16.265198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:16.265198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:16.265198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:16.265198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:16.265198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:16.265198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:16.265198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:16.265198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:16.265198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:16.265198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:16.265198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:16.265198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:16.265198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:16.265198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:16.265198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:16.266235 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:16.266235 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:16.266235 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:16.268217 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 231/466] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:16.269633 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7c5f49-4a19-4391-9f1c-0188dfa41f86 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:16.270050 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7c5f49-4a19-4391-9f1c-0188dfa41f86 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:16.270445 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7c5f49-4a19-4391-9f1c-0188dfa41f86 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:16.271822 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 232/467] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:16.272681 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7c5f49-4a19-4391-9f1c-0188dfa41f86 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:16.368180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98cf11aa-0bc7-4e1d-bd64-ef0613121836 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:16.402907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7c5f49-4a19-4391-9f1c-0188dfa41f86 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:16.407199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff04ada8-51a8-41a7-be14-32e6b99b8947 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:16.407911 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff04ada8-51a8-41a7-be14-32e6b99b8947 None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:16.408295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff04ada8-51a8-41a7-be14-32e6b99b8947 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:16.408669 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff04ada8-51a8-41a7-be14-32e6b99b8947 None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:16.409419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff04ada8-51a8-41a7-be14-32e6b99b8947 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:16.411024 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff04ada8-51a8-41a7-be14-32e6b99b8947 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:16.430505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:16.446713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fcbbe8-c529-4210-acb2-d94aacbaf011 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:16.447341 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fcbbe8-c529-4210-acb2-d94aacbaf011 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:16.447628 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fcbbe8-c529-4210-acb2-d94aacbaf011 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:16.447897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fcbbe8-c529-4210-acb2-d94aacbaf011 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:16.448664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fcbbe8-c529-4210-acb2-d94aacbaf011 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:16.450104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fcbbe8-c529-4210-acb2-d94aacbaf011 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:16.459336 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:16.507333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 236/468] 10.222.0.39 () {60 vars in 1132 bytes} [Fri Sep 30 09:27:16 2022] POST /identity/v3/users => generated 246 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:27:16.508398 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff04ada8-51a8-41a7-be14-32e6b99b8947 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:16.530682 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 233/469] 10.222.0.39 () {60 vars in 1132 bytes} [Fri Sep 30 09:27:16 2022] POST /identity/v3/users => generated 248 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:27:16.532380 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4bd680-0eca-41c1-8e15-42f76c88f112 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:16.532721 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4bd680-0eca-41c1-8e15-42f76c88f112 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:16.533072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4bd680-0eca-41c1-8e15-42f76c88f112 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:16.557569 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-1b4bd680-0eca-41c1-8e15-42f76c88f112 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:16.559388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1b4bd680-0eca-41c1-8e15-42f76c88f112 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:16.650018 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff04ada8-51a8-41a7-be14-32e6b99b8947 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:16.655883 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 234/470] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:16.655883 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c49405-9664-4589-8d79-9886c45ce116 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:16.655883 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c49405-9664-4589-8d79-9886c45ce116 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:16.655883 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c49405-9664-4589-8d79-9886c45ce116 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:16.655883 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 235/471] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:16.663346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202847dd-4985-4cca-905b-c65fe2bfbcce None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:16.663346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202847dd-4985-4cca-905b-c65fe2bfbcce None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:16.663346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202847dd-4985-4cca-905b-c65fe2bfbcce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:16.700384 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-202847dd-4985-4cca-905b-c65fe2bfbcce None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:16.701734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-202847dd-4985-4cca-905b-c65fe2bfbcce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:16.729228 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdbba18-f326-48a4-a895-58078f9ab06b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:16.729853 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdbba18-f326-48a4-a895-58078f9ab06b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:16.730137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdbba18-f326-48a4-a895-58078f9ab06b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:16.730413 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdbba18-f326-48a4-a895-58078f9ab06b None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:16.731104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdbba18-f326-48a4-a895-58078f9ab06b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:16.731412 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdbba18-f326-48a4-a895-58078f9ab06b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:16.733836 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdbba18-f326-48a4-a895-58078f9ab06b None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:16.749460 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 237/472] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:16.754899 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdbba18-f326-48a4-a895-58078f9ab06b None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:16.784649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 236/473] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:16.830255 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:16.862741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdbba18-f326-48a4-a895-58078f9ab06b None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:16.899762 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3620d5d-8432-4722-b3d0-6f5997731477 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:16.900429 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3620d5d-8432-4722-b3d0-6f5997731477 None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:16.900647 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3620d5d-8432-4722-b3d0-6f5997731477 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:16.900915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3620d5d-8432-4722-b3d0-6f5997731477 None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:16.901557 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3620d5d-8432-4722-b3d0-6f5997731477 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:16.903032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3620d5d-8432-4722-b3d0-6f5997731477 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:16.956671 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202847dd-4985-4cca-905b-c65fe2bfbcce None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:17.007638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 238/474] 10.222.0.39 () {60 vars in 1132 bytes} [Fri Sep 30 09:27:16 2022] POST /identity/v3/users => generated 248 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:27:17.012361 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:17.077587 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202847dd-4985-4cca-905b-c65fe2bfbcce None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:17.114794 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3620d5d-8432-4722-b3d0-6f5997731477 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:17.133095 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01402c0e-f37e-4d69-80ba-aa4dd9c59681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:17.133753 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01402c0e-f37e-4d69-80ba-aa4dd9c59681 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:17.134026 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01402c0e-f37e-4d69-80ba-aa4dd9c59681 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:17.134292 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01402c0e-f37e-4d69-80ba-aa4dd9c59681 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:17.155534 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01402c0e-f37e-4d69-80ba-aa4dd9c59681 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:17.157519 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01402c0e-f37e-4d69-80ba-aa4dd9c59681 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:17.203760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-01402c0e-f37e-4d69-80ba-aa4dd9c59681 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:17.203760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:17.203760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:17.203760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:17.203760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:17.203760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:17.203760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:17.203760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:17.203760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:17.203760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:17.203760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:17.203760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:17.203760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:17.203760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:17.203760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:17.203760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:17.204705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:17.204705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:17.204705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:17.204705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:17.204705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:17.204705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:17.204705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:17.204705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:17.204705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:17.204705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:17.204705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:17.204705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:17.204705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:17.204705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:17.204705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:17.204705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:17.204705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:17.205593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:17.205593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:17.205593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:17.205593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 237/475] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:17.211152 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01402c0e-f37e-4d69-80ba-aa4dd9c59681 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:17.261077 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3620d5d-8432-4722-b3d0-6f5997731477 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:17.275703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:17.298756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9a8d7e-9b50-4a1a-8d97-4dd6911f7ffc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:17.299703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9a8d7e-9b50-4a1a-8d97-4dd6911f7ffc None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:17.300184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9a8d7e-9b50-4a1a-8d97-4dd6911f7ffc None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:17.300670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9a8d7e-9b50-4a1a-8d97-4dd6911f7ffc None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:17.301565 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9a8d7e-9b50-4a1a-8d97-4dd6911f7ffc None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:17.303198 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9a8d7e-9b50-4a1a-8d97-4dd6911f7ffc None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:17.335448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01402c0e-f37e-4d69-80ba-aa4dd9c59681 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:17.336328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.sql.core [None req-8e9a8d7e-9b50-4a1a-8d97-4dd6911f7ffc None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 30 09:27:17.336328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 30 09:27:17.336328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [parameters: {'id': '84ee51fbceef4b7f817a4b6258e81726', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 30 09:27:17.336328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81685) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 30 09:27:17.339198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:17.354234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-8e9a8d7e-9b50-4a1a-8d97-4dd6911f7ffc None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 30 09:27:17.354234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:17.354234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 30 09:27:17.354234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 30 09:27:17.354234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 30 09:27:17.354234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 30 09:27:17.354234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 30 09:27:17.354234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application result = self._query(query) Sep 30 09:27:17.354234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 30 09:27:17.354234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application conn.query(q) Sep 30 09:27:17.354234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 30 09:27:17.354234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 30 09:27:17.354234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 30 09:27:17.354234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application result.read() Sep 30 09:27:17.354234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 30 09:27:17.355219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 30 09:27:17.355219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 30 09:27:17.355219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 30 09:27:17.355219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 30 09:27:17.355219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 30 09:27:17.355219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 30 09:27:17.355219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 30 09:27:17.355219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 30 09:27:17.355219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:17.355219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 30 09:27:17.355219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:17.355219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:17.355219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 30 09:27:17.355219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 30 09:27:17.355219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 30 09:27:17.355219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return ref.to_dict() Sep 30 09:27:17.355219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 30 09:27:17.355219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application next(self.gen) Sep 30 09:27:17.356172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 30 09:27:17.356172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application yield resource Sep 30 09:27:17.356172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 30 09:27:17.356172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application next(self.gen) Sep 30 09:27:17.356172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 30 09:27:17.356172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self.session.rollback() Sep 30 09:27:17.356172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 30 09:27:17.356172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self.force_reraise() Sep 30 09:27:17.356172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 30 09:27:17.356172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise self.value Sep 30 09:27:17.356172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 30 09:27:17.356172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 30 09:27:17.356172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 30 09:27:17.356172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application session.commit() Sep 30 09:27:17.356172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 30 09:27:17.356172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 30 09:27:17.356172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 30 09:27:17.357048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._prepare_impl() Sep 30 09:27:17.357048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 30 09:27:17.357048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self.session.flush() Sep 30 09:27:17.357048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 30 09:27:17.357048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._flush(objects) Sep 30 09:27:17.357048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 30 09:27:17.357048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 30 09:27:17.357048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 30 09:27:17.357048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application compat.raise_( Sep 30 09:27:17.357048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 30 09:27:17.357048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception Sep 30 09:27:17.357048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 30 09:27:17.357048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application flush_context.execute() Sep 30 09:27:17.357048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 30 09:27:17.357048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rec.execute(self) Sep 30 09:27:17.357048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 30 09:27:17.357048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 30 09:27:17.365393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 30 09:27:17.365393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 30 09:27:17.365393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 30 09:27:17.365393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 30 09:27:17.365393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 30 09:27:17.365393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 30 09:27:17.365393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 30 09:27:17.365393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 30 09:27:17.365393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 30 09:27:17.365393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 30 09:27:17.365393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 30 09:27:17.365393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 30 09:27:17.365393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 30 09:27:17.365393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 30 09:27:17.365393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 30 09:27:17.365393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception Sep 30 09:27:17.367691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 30 09:27:17.367691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 30 09:27:17.367691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 30 09:27:17.367691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 30 09:27:17.367691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 30 09:27:17.367691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application result = self._query(query) Sep 30 09:27:17.367691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 30 09:27:17.367691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application conn.query(q) Sep 30 09:27:17.367691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 30 09:27:17.367691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 30 09:27:17.367691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 30 09:27:17.367691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application result.read() Sep 30 09:27:17.367691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 30 09:27:17.367691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 30 09:27:17.367691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 30 09:27:17.367691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 30 09:27:17.369613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 30 09:27:17.369613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 30 09:27:17.369613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 30 09:27:17.369613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 30 09:27:17.369613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 30 09:27:17.369613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 30 09:27:17.369613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [parameters: {'id': '84ee51fbceef4b7f817a4b6258e81726', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 30 09:27:17.369613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 30 09:27:17.369613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:17.369613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:27:17.369613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:17.369613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:17.369613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:17.369613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:17.369613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:17.369613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:17.371728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:17.371728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:17.371728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:17.371728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:17.371728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:17.371728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:17.371728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 30 09:27:17.371728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 30 09:27:17.371728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:17.371728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:17.371728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 30 09:27:17.371728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 30 09:27:17.371728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 30 09:27:17.371728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 30 09:27:17.371728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 30 09:27:17.371728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:17.374714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 239/476] 10.222.0.39 () {60 vars in 1131 bytes} [Fri Sep 30 09:27:17 2022] POST /identity/v3/roles => generated 144 bytes in 242 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:17.374714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9a8d7e-9b50-4a1a-8d97-4dd6911f7ffc None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:17.380742 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591d973f-1991-4692-86ba-3f3ee4309624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:17.381606 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591d973f-1991-4692-86ba-3f3ee4309624 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:17.382358 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591d973f-1991-4692-86ba-3f3ee4309624 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:17.382853 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591d973f-1991-4692-86ba-3f3ee4309624 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:17.384324 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591d973f-1991-4692-86ba-3f3ee4309624 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:17.384938 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591d973f-1991-4692-86ba-3f3ee4309624 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:17.387118 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591d973f-1991-4692-86ba-3f3ee4309624 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:17.405755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 238/477] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:17.412703 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591d973f-1991-4692-86ba-3f3ee4309624 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:17.521820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9a8d7e-9b50-4a1a-8d97-4dd6911f7ffc None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:17.526400 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591d973f-1991-4692-86ba-3f3ee4309624 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:17.565020 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b7d2b6-1c92-43b3-a4f7-1707033a2644 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:17.566205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b7d2b6-1c92-43b3-a4f7-1707033a2644 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:17.566690 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b7d2b6-1c92-43b3-a4f7-1707033a2644 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:17.567201 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b7d2b6-1c92-43b3-a4f7-1707033a2644 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:17.567573 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca802ec3-0f47-4a7e-a491-7a16e3276e2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:17.568245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b7d2b6-1c92-43b3-a4f7-1707033a2644 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:17.568366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca802ec3-0f47-4a7e-a491-7a16e3276e2c None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:17.568600 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca802ec3-0f47-4a7e-a491-7a16e3276e2c None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:17.568937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca802ec3-0f47-4a7e-a491-7a16e3276e2c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:17.570520 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b7d2b6-1c92-43b3-a4f7-1707033a2644 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:17.580455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca802ec3-0f47-4a7e-a491-7a16e3276e2c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:17.590616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-ca802ec3-0f47-4a7e-a491-7a16e3276e2c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:27:17.590616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:17.590616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:17.590616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:17.590616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:17.590616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:17.590616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:17.590616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:17.590616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:17.590616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:17.590616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:17.590616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:17.590616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:17.590616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:17.590616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:17.590616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:17.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:17.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:17.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:17.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:17.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:17.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:17.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:17.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:17.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:17.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:17.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:17.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:17.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:17.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:17.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:17.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:17.592916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:17.592916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:17.592916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:17.592916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:17.592916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:17.592916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:17.592916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:17.592916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:17.592916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:17.592916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:17.592916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 30 09:27:17.592916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:17.593529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 240/478] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:27:17 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:17.594956 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca802ec3-0f47-4a7e-a491-7a16e3276e2c None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:17.633584 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:17.683200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 239/479] 10.222.0.39 () {60 vars in 1132 bytes} [Fri Sep 30 09:27:17 2022] POST /identity/v3/users => generated 249 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:27:17.717587 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca802ec3-0f47-4a7e-a491-7a16e3276e2c None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:17.757064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75aa4759-28f8-4967-987f-d0bcda6a60c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:17.757708 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75aa4759-28f8-4967-987f-d0bcda6a60c4 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:17.758204 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75aa4759-28f8-4967-987f-d0bcda6a60c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:17.758671 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75aa4759-28f8-4967-987f-d0bcda6a60c4 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:17.759340 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75aa4759-28f8-4967-987f-d0bcda6a60c4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:17.759705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75aa4759-28f8-4967-987f-d0bcda6a60c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:17.761233 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75aa4759-28f8-4967-987f-d0bcda6a60c4 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:17.774847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 241/480] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:17 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:19.316181 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bc56b9-cac1-4510-a48e-b37c61c971ad None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:19.316697 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bc56b9-cac1-4510-a48e-b37c61c971ad None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:19.316772 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bc56b9-cac1-4510-a48e-b37c61c971ad None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:19.318300 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 240/481] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:19.323509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5971eaa-71d5-47f9-b0fe-d3c1c824a39c None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:19.323904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5971eaa-71d5-47f9-b0fe-d3c1c824a39c None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:19.324088 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5971eaa-71d5-47f9-b0fe-d3c1c824a39c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:19.345198 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-d5971eaa-71d5-47f9-b0fe-d3c1c824a39c None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:19.346389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d5971eaa-71d5-47f9-b0fe-d3c1c824a39c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:19.409110 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57aa7d5c-483b-41e0-9e03-e79c8fe0607f None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:19.409415 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57aa7d5c-483b-41e0-9e03-e79c8fe0607f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:19.409691 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57aa7d5c-483b-41e0-9e03-e79c8fe0607f None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:19.411313 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 241/482] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:19.425503 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79a6aaa-0f37-43d1-9120-839121f9a816 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:19.425823 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79a6aaa-0f37-43d1-9120-839121f9a816 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:19.426069 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79a6aaa-0f37-43d1-9120-839121f9a816 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:19.439449 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 242/483] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:19.468102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d79a6aaa-0f37-43d1-9120-839121f9a816 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:19.469609 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d79a6aaa-0f37-43d1-9120-839121f9a816 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:19.489723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:19.543692 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 242/484] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:19.762998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f353be-00d2-4934-843f-f735aff0d203 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:19.763707 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f353be-00d2-4934-843f-f735aff0d203 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:19.764182 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f353be-00d2-4934-843f-f735aff0d203 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:19.765948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 243/485] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:19.770935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1773212c-089e-4f48-b972-542c359a4f6c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:19.771471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1773212c-089e-4f48-b972-542c359a4f6c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:19.771949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1773212c-089e-4f48-b972-542c359a4f6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:19.790521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-1773212c-089e-4f48-b972-542c359a4f6c None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:19.791975 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1773212c-089e-4f48-b972-542c359a4f6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:19.859354 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 243/486] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:19.894251 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f353be-00d2-4934-843f-f735aff0d203 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:19.931646 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:19.993157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1773212c-089e-4f48-b972-542c359a4f6c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:20.079900 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f353be-00d2-4934-843f-f735aff0d203 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:20.191464 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:20.222785 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e839f42b-84e7-4d04-877d-d23630f9922d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:20.223420 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e839f42b-84e7-4d04-877d-d23630f9922d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:20.223704 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e839f42b-84e7-4d04-877d-d23630f9922d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:20.224065 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e839f42b-84e7-4d04-877d-d23630f9922d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:20.226193 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1773212c-089e-4f48-b972-542c359a4f6c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:20.235521 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e839f42b-84e7-4d04-877d-d23630f9922d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:20.240653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-e839f42b-84e7-4d04-877d-d23630f9922d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:20.240653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:20.240653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:20.240653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:20.240653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:20.240653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:20.240653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:20.240653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:20.240653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:20.240653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:20.240653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:20.240653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:20.240653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:20.240653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:20.240653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:20.240653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:20.241578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:20.241578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:20.241578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:20.241578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:20.241578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:20.241578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:20.241578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:20.241578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:20.241578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:20.241578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:20.241578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:20.241578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:20.241578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:20.241578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:20.241578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:20.241578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:20.242393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:20.242393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:20.242393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:20.242393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:20.242393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:20.242393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:20.242393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:20.242393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:20.242393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:20.242393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:20.242393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:20.242393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:20.242393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 244/487] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 348 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:20.243108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e839f42b-84e7-4d04-877d-d23630f9922d None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:20.271857 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9c0f24-aaba-48ae-9802-12ce546222bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:20.272490 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9c0f24-aaba-48ae-9802-12ce546222bf None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:20.273032 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9c0f24-aaba-48ae-9802-12ce546222bf None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:20.273309 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9c0f24-aaba-48ae-9802-12ce546222bf None admin] PATH_INFO: `/v3/roles/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:20.285872 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9c0f24-aaba-48ae-9802-12ce546222bf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:20.291008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-1e9c0f24-aaba-48ae-9802-12ce546222bf None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:20.291008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:20.291008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:20.291008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:20.291008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:20.291008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:20.291008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:20.291008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:20.291008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:20.291008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:20.291008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:20.291008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:20.291008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:20.291008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:20.291008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:20.291008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:20.291894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:20.291894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:20.291894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:20.291894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:20.291894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:20.291894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:20.291894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:20.291894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:20.291894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:20.291894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:20.291894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:20.291894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:20.291894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:20.291894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:20.291894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:20.291894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:20.292797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:20.292797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:20.292797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:20.292797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:20.292797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:20.292797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:20.292797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:20.292797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:20.292797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:20.292797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:20.292797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:20.292797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:20.292797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 244/488] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:20.293466 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9c0f24-aaba-48ae-9802-12ce546222bf None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:20.354020 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:20.367605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e839f42b-84e7-4d04-877d-d23630f9922d None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:20.413407 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4cfca31-c4b9-486f-8a0c-c13d11d785cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:20.414079 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cfca31-c4b9-486f-8a0c-c13d11d785cb None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:20.414357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cfca31-c4b9-486f-8a0c-c13d11d785cb None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:20.414648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cfca31-c4b9-486f-8a0c-c13d11d785cb None admin] PATH_INFO: `/v3/roles/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:20.421566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9c0f24-aaba-48ae-9802-12ce546222bf None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:20.430395 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4cfca31-c4b9-486f-8a0c-c13d11d785cb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:20.438097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-d4cfca31-c4b9-486f-8a0c-c13d11d785cb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:20.438097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:20.438097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:20.438097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:20.438097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:20.438097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:20.438097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:20.438097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:20.438097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:20.438097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:20.438097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:20.438097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:20.438097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:20.438097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:20.438097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:20.438097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:20.439139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:20.439139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:20.439139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:20.439139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:20.439139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:20.439139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:20.439139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:20.439139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:20.439139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:20.439139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:20.439139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:20.439139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:20.439139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:20.439139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:20.439139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:20.439139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:20.445030 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:20.445030 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:20.445030 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:20.445030 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:20.445030 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:20.445030 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:20.445030 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:20.445030 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:20.445030 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:20.445030 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:20.445030 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:20.445030 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:20.445030 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 245/489] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:20.445030 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4cfca31-c4b9-486f-8a0c-c13d11d785cb None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:20.470013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91c924f-1fab-459e-b669-747c68837885 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:20.470659 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91c924f-1fab-459e-b669-747c68837885 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:20.470937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91c924f-1fab-459e-b669-747c68837885 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:20.471203 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91c924f-1fab-459e-b669-747c68837885 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:20.471855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d91c924f-1fab-459e-b669-747c68837885 None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:20.472152 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d91c924f-1fab-459e-b669-747c68837885 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:20.473514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d91c924f-1fab-459e-b669-747c68837885 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:20.482993 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 245/490] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:27:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:20.484038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91c924f-1fab-459e-b669-747c68837885 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:20.486687 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:20.556226 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4cfca31-c4b9-486f-8a0c-c13d11d785cb None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:20.586950 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91c924f-1fab-459e-b669-747c68837885 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:20.588452 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798e208e-b44a-4965-8b1c-4418c2629dd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:20.589046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798e208e-b44a-4965-8b1c-4418c2629dd6 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:20.589333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798e208e-b44a-4965-8b1c-4418c2629dd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:20.589601 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798e208e-b44a-4965-8b1c-4418c2629dd6 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:20.590256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798e208e-b44a-4965-8b1c-4418c2629dd6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:20.590558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798e208e-b44a-4965-8b1c-4418c2629dd6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:20.592489 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798e208e-b44a-4965-8b1c-4418c2629dd6 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:20.600251 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 246/491] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:27:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:20.601653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c79ce9-6e9e-489c-9912-d861f9283ad8 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:20.601941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c79ce9-6e9e-489c-9912-d861f9283ad8 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:20.602215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c79ce9-6e9e-489c-9912-d861f9283ad8 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:20.603421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 247/492] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:20 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:20.604158 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c79ce9-6e9e-489c-9912-d861f9283ad8 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:20.617316 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:20.625690 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe23c80-7e5e-47f2-a347-887768819442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:20.633806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe23c80-7e5e-47f2-a347-887768819442 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:20.633806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe23c80-7e5e-47f2-a347-887768819442 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:20.633806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe23c80-7e5e-47f2-a347-887768819442 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:20.633806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe23c80-7e5e-47f2-a347-887768819442 None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:20.633806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe23c80-7e5e-47f2-a347-887768819442 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:20.633806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe23c80-7e5e-47f2-a347-887768819442 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:20.637475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 246/493] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:27:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:20.638760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604a6b55-0a6a-4331-a118-db55db79a812 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:20.639045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604a6b55-0a6a-4331-a118-db55db79a812 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:20.639316 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604a6b55-0a6a-4331-a118-db55db79a812 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:20.640721 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 247/494] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:20 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:20.641268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604a6b55-0a6a-4331-a118-db55db79a812 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:20.702483 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:20.707633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c79ce9-6e9e-489c-9912-d861f9283ad8 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:20.741125 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604a6b55-0a6a-4331-a118-db55db79a812 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:20.741593 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06fa054-4719-4345-acc8-f8635526b0dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:20.742211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06fa054-4719-4345-acc8-f8635526b0dc None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:20.742496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06fa054-4719-4345-acc8-f8635526b0dc None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:20.742796 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06fa054-4719-4345-acc8-f8635526b0dc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:20.752559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06fa054-4719-4345-acc8-f8635526b0dc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:20.754007 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06fa054-4719-4345-acc8-f8635526b0dc None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:20.768502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-b06fa054-4719-4345-acc8-f8635526b0dc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:20.768502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:20.768502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:20.768502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:20.768502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:20.768502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:20.768502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:20.768502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:20.768502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:20.768502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:20.768502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:20.768502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:20.768502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:20.768502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:20.768502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:20.768502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:20.769487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:20.769487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:20.769487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:20.769487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:20.769487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:20.769487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:20.769487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:20.769487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:20.769487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:20.769487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:20.769487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:20.769487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:20.769487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:20.769487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:20.769487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:20.769487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:20.769487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:20.770391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:20.770391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:20.770391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:20.770391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 248/495] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:20.770736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ddcfd9-e1d5-48db-9782-d0170da25bc4 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:20.770979 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ddcfd9-e1d5-48db-9782-d0170da25bc4 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:20.771247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ddcfd9-e1d5-48db-9782-d0170da25bc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:20.776191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac8f41d-b98d-4a9a-b0c8-75b90a9a8293 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:20.776822 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac8f41d-b98d-4a9a-b0c8-75b90a9a8293 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:20.777099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac8f41d-b98d-4a9a-b0c8-75b90a9a8293 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:20.777367 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac8f41d-b98d-4a9a-b0c8-75b90a9a8293 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:20.788585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac8f41d-b98d-4a9a-b0c8-75b90a9a8293 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:20.788854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-b1ddcfd9-e1d5-48db-9782-d0170da25bc4 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:20.790010 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac8f41d-b98d-4a9a-b0c8-75b90a9a8293 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:20.790104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b1ddcfd9-e1d5-48db-9782-d0170da25bc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:20.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-3ac8f41d-b98d-4a9a-b0c8-75b90a9a8293 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:20.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:20.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:20.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:20.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:20.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:20.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:20.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:20.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:20.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:20.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:20.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:20.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:20.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:20.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:20.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:20.805723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:20.805723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:20.805723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:20.805723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:20.805723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:20.805723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:20.805723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:20.805723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:20.805723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:20.805723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:20.805723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:20.805723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:20.805723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:20.805723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:20.805723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:20.805723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:20.805723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:20.806621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:20.806621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:20.806621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:20.806621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 248/496] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:20.806621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac8f41d-b98d-4a9a-b0c8-75b90a9a8293 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:20.852840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:20.862688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 249/497] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:20.863831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897b90fb-696a-47b3-8a55-27459ecc1051 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:20.864119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897b90fb-696a-47b3-8a55-27459ecc1051 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:20.864390 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897b90fb-696a-47b3-8a55-27459ecc1051 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:20.883193 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-897b90fb-696a-47b3-8a55-27459ecc1051 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:20.884369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-897b90fb-696a-47b3-8a55-27459ecc1051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:20.921485 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac8f41d-b98d-4a9a-b0c8-75b90a9a8293 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:20.939289 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:20.967179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 250/498] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:20.967179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897b90fb-696a-47b3-8a55-27459ecc1051 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:20.969452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0b3294-e0bd-43f1-b5d3-042361f1fc78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:20.970205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0b3294-e0bd-43f1-b5d3-042361f1fc78 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:20.970547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0b3294-e0bd-43f1-b5d3-042361f1fc78 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:20.970930 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0b3294-e0bd-43f1-b5d3-042361f1fc78 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:20.982065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0b3294-e0bd-43f1-b5d3-042361f1fc78 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:20.983866 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0b3294-e0bd-43f1-b5d3-042361f1fc78 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:21.000255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-4a0b3294-e0bd-43f1-b5d3-042361f1fc78 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:21.000255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:21.000255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:21.000255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:21.000255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:21.000255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:21.000255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:21.000255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:21.000255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:21.000255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:21.000255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:21.000255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:21.000255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:21.000255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:21.000255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:21.000255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:21.001290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:21.001290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:21.001290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:21.001290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:21.001290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:21.001290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:21.001290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:21.001290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:21.001290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:21.001290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:21.001290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:21.001290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:21.001290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:21.001290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:21.001290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:21.001290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:21.001290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:21.002171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:21.002171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:21.002171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:21.006271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 249/499] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:21.007425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0b3294-e0bd-43f1-b5d3-042361f1fc78 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:21.073903 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897b90fb-696a-47b3-8a55-27459ecc1051 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:21.100965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:21.114006 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d314d33-3783-4201-96c6-8b8a0265dcdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:21.114986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d314d33-3783-4201-96c6-8b8a0265dcdf None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:21.115564 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d314d33-3783-4201-96c6-8b8a0265dcdf None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:21.115846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d314d33-3783-4201-96c6-8b8a0265dcdf None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:21.116491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d314d33-3783-4201-96c6-8b8a0265dcdf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:21.116796 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d314d33-3783-4201-96c6-8b8a0265dcdf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:21.118661 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d314d33-3783-4201-96c6-8b8a0265dcdf None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:21.122116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0b3294-e0bd-43f1-b5d3-042361f1fc78 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:21.133932 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 251/500] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:21.139768 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d314d33-3783-4201-96c6-8b8a0265dcdf None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:21.159932 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0751de7-ad7d-44bd-b61a-046523ab6545 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:21.160542 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0751de7-ad7d-44bd-b61a-046523ab6545 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:21.160825 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0751de7-ad7d-44bd-b61a-046523ab6545 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:21.161100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0751de7-ad7d-44bd-b61a-046523ab6545 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:21.161764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0751de7-ad7d-44bd-b61a-046523ab6545 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:21.162069 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0751de7-ad7d-44bd-b61a-046523ab6545 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:21.163504 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0751de7-ad7d-44bd-b61a-046523ab6545 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:21.176055 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 250/501] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:21.181046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0751de7-ad7d-44bd-b61a-046523ab6545 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:21.212983 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:21.252370 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d314d33-3783-4201-96c6-8b8a0265dcdf None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:21.286116 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f85027-b15b-472a-9cb5-31c12759b059 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:21.286753 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f85027-b15b-472a-9cb5-31c12759b059 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:21.287035 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f85027-b15b-472a-9cb5-31c12759b059 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:21.287128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0751de7-ad7d-44bd-b61a-046523ab6545 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:21.287307 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f85027-b15b-472a-9cb5-31c12759b059 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:21.287936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f85027-b15b-472a-9cb5-31c12759b059 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:21.288235 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f85027-b15b-472a-9cb5-31c12759b059 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:21.289602 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f85027-b15b-472a-9cb5-31c12759b059 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:21.300934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 252/502] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:21.301710 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f85027-b15b-472a-9cb5-31c12759b059 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:21.325393 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00421b48-c87e-4c17-843f-d410d3eb2a33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:21.326576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00421b48-c87e-4c17-843f-d410d3eb2a33 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:21.326888 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00421b48-c87e-4c17-843f-d410d3eb2a33 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:21.327163 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00421b48-c87e-4c17-843f-d410d3eb2a33 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:21.337553 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00421b48-c87e-4c17-843f-d410d3eb2a33 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:21.339044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00421b48-c87e-4c17-843f-d410d3eb2a33 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:21.343445 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:21.353277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-00421b48-c87e-4c17-843f-d410d3eb2a33 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:21.353277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:21.353277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:21.353277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:21.353277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:21.353277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:21.353277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:21.353277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:21.353277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:21.353277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:21.353277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:21.353277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:21.353277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:21.353277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:21.353277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:21.353277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:21.359310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:21.359310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:21.359310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:21.359310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:21.359310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:21.359310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:21.359310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:21.359310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:21.359310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:21.359310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:21.359310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:21.359310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:21.359310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:21.359310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:21.359310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:21.359310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:21.359310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:21.360200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:21.360200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:21.360200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:21.360200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 251/503] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:21.360200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00421b48-c87e-4c17-843f-d410d3eb2a33 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:21.400054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f85027-b15b-472a-9cb5-31c12759b059 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:21.432269 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3ad695-0bf0-4f7a-8411-5edb45257516 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:21.432892 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3ad695-0bf0-4f7a-8411-5edb45257516 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:21.433173 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3ad695-0bf0-4f7a-8411-5edb45257516 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:21.433444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3ad695-0bf0-4f7a-8411-5edb45257516 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:21.440793 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3ad695-0bf0-4f7a-8411-5edb45257516 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:21.440940 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:21.442243 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3ad695-0bf0-4f7a-8411-5edb45257516 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:21.455421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-1d3ad695-0bf0-4f7a-8411-5edb45257516 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 30 09:27:21.455421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:21.455421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:21.455421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:21.455421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:21.455421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:21.455421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:21.455421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:21.455421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:21.455421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:21.455421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:21.455421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:21.455421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:21.455421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:21.455421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:21.455421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:21.458540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:21.458540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:21.458540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:21.458540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:21.458540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:21.458540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:21.458540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:21.458540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:21.458540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:21.458540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:21.458540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:21.458540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:21.458540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:21.458540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:21.458540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:21.458540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:21.458540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:21.458540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:21.459532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:21.459532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:21.459532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:21.459532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:21.459532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:21.459532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:21.459532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:21.459532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:21.459532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 30 09:27:21.459532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:21.459532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 253/504] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:27:21 2022] GET /identity/v3/users/glance => generated 84 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:21.459532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3ad695-0bf0-4f7a-8411-5edb45257516 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:21.464913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00421b48-c87e-4c17-843f-d410d3eb2a33 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:21.500001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e5dc5b-7810-4854-b237-375c82d548fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:21.500617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e5dc5b-7810-4854-b237-375c82d548fc None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:21.500898 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e5dc5b-7810-4854-b237-375c82d548fc None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:21.501164 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e5dc5b-7810-4854-b237-375c82d548fc None admin] PATH_INFO: `/v3/users/nova` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:21.508181 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e5dc5b-7810-4854-b237-375c82d548fc None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:21.509597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e5dc5b-7810-4854-b237-375c82d548fc None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:21.522784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-05e5dc5b-7810-4854-b237-375c82d548fc None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 30 09:27:21.522784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:21.522784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:21.522784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:21.522784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:21.522784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:21.522784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:21.522784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:21.522784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:21.522784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:21.522784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:21.522784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:21.522784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:21.522784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:21.522784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:21.522784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:21.523698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:21.523698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:21.523698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:21.523698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:21.523698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:21.523698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:21.523698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:21.523698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:21.523698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:21.523698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:21.523698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:21.523698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:21.523698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:21.523698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:21.523698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:21.523698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:21.523698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:21.523698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:21.524617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:21.524617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:21.524617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:21.524617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:21.524617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:21.524617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:21.524617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:21.524617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:21.524617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 30 09:27:21.524617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:21.524617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 252/505] 10.222.0.39 () {56 vars in 1093 bytes} [Fri Sep 30 09:27:21 2022] GET /identity/v3/users/nova => generated 82 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:21.525199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e5dc5b-7810-4854-b237-375c82d548fc None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:21.561007 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3ad695-0bf0-4f7a-8411-5edb45257516 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:21.562501 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:21.595013 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15a4180-f24d-49d8-9031-9178bd4248a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:21.595665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15a4180-f24d-49d8-9031-9178bd4248a2 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:21.596202 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15a4180-f24d-49d8-9031-9178bd4248a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:21.596528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15a4180-f24d-49d8-9031-9178bd4248a2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:21.606680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15a4180-f24d-49d8-9031-9178bd4248a2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:21.611968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-f15a4180-f24d-49d8-9031-9178bd4248a2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:21.611968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:21.611968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:21.611968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:21.611968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:21.611968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:21.611968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:21.611968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:21.611968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:21.611968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:21.611968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:21.611968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:21.611968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:21.611968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:21.611968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:21.611968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:21.612910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:21.612910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:21.612910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:21.612910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:21.612910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:21.612910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:21.612910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:21.612910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:21.612910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:21.612910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:21.612910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:21.612910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:21.612910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:21.612910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:21.612910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:21.612910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:21.613812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:21.613812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:21.613812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:21.613812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:21.613812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:21.613812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:21.613812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:21.613812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:21.613812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:21.613812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:21.613812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:21.613812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:21.613812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 254/506] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:21 2022] GET /identity/v3/roles/service => generated 85 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:21.614497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15a4180-f24d-49d8-9031-9178bd4248a2 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:21.624307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e5dc5b-7810-4854-b237-375c82d548fc None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:21.656977 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef97be7f-db7e-4df6-b331-d1746cc4f6b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:21.657708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef97be7f-db7e-4df6-b331-d1746cc4f6b4 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:21.657996 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef97be7f-db7e-4df6-b331-d1746cc4f6b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:21.658273 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef97be7f-db7e-4df6-b331-d1746cc4f6b4 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:21.665760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef97be7f-db7e-4df6-b331-d1746cc4f6b4 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:21.667256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef97be7f-db7e-4df6-b331-d1746cc4f6b4 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:21.670243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:21.679059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-ef97be7f-db7e-4df6-b331-d1746cc4f6b4 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 30 09:27:21.679059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:21.679059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:21.679059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:21.679059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:21.679059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:21.679059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:21.679059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:21.679059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:21.679059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:21.679059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:21.679059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:21.679059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:21.679059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:21.679059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:21.679059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:21.687766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:21.687766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:21.687766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:21.687766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:21.687766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:21.687766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:21.687766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:21.687766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:21.687766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:21.687766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:21.687766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:21.687766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:21.687766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:21.687766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:21.687766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:21.687766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:21.687766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:21.687766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:21.688707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:21.688707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:21.688707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:21.688707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:21.688707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:21.688707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:21.688707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:21.688707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:21.688707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 30 09:27:21.688707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:21.688707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 253/507] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:27:21 2022] GET /identity/v3/users/cinder => generated 84 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:21.688707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef97be7f-db7e-4df6-b331-d1746cc4f6b4 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:21.715392 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15a4180-f24d-49d8-9031-9178bd4248a2 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:21.749137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b44e6b2-953c-443a-aa63-3d0909603aaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:21.749734 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b44e6b2-953c-443a-aa63-3d0909603aaa None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:21.750015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b44e6b2-953c-443a-aa63-3d0909603aaa None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:21.750279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b44e6b2-953c-443a-aa63-3d0909603aaa None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:21.750939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b44e6b2-953c-443a-aa63-3d0909603aaa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:21.751237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b44e6b2-953c-443a-aa63-3d0909603aaa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:21.752645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b44e6b2-953c-443a-aa63-3d0909603aaa None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:21.764153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 255/508] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:21.765003 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b44e6b2-953c-443a-aa63-3d0909603aaa None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:21.784012 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef97be7f-db7e-4df6-b331-d1746cc4f6b4 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:21.788898 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:21.817370 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da68f8b7-07ee-4d19-8695-ce6f14d05f35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:21.817968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da68f8b7-07ee-4d19-8695-ce6f14d05f35 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:21.818247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da68f8b7-07ee-4d19-8695-ce6f14d05f35 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:21.818515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da68f8b7-07ee-4d19-8695-ce6f14d05f35 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:21.819257 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da68f8b7-07ee-4d19-8695-ce6f14d05f35 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:21.819563 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da68f8b7-07ee-4d19-8695-ce6f14d05f35 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:21.820999 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da68f8b7-07ee-4d19-8695-ce6f14d05f35 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:21.855989 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 254/509] 10.222.0.39 () {56 vars in 1137 bytes} [Fri Sep 30 09:27:21 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:21.856790 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da68f8b7-07ee-4d19-8695-ce6f14d05f35 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:21.861866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b44e6b2-953c-443a-aa63-3d0909603aaa None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:21.896056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50efd386-f692-49b2-b595-f61d9b2b1881 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:21.896650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50efd386-f692-49b2-b595-f61d9b2b1881 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:21.896927 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50efd386-f692-49b2-b595-f61d9b2b1881 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:21.897196 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50efd386-f692-49b2-b595-f61d9b2b1881 None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:21.897902 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50efd386-f692-49b2-b595-f61d9b2b1881 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:21.898200 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50efd386-f692-49b2-b595-f61d9b2b1881 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:21.899663 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50efd386-f692-49b2-b595-f61d9b2b1881 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:21.916455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:21.935335 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 256/510] 10.222.0.39 () {56 vars in 1133 bytes} [Fri Sep 30 09:27:21 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:21.936627 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50efd386-f692-49b2-b595-f61d9b2b1881 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:21.958811 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da68f8b7-07ee-4d19-8695-ce6f14d05f35 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:21.991952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e65ba4-e953-4307-bc38-88a6d2f47e50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:21.992559 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e65ba4-e953-4307-bc38-88a6d2f47e50 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:21.992842 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e65ba4-e953-4307-bc38-88a6d2f47e50 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:21.993113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e65ba4-e953-4307-bc38-88a6d2f47e50 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:21.993768 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e65ba4-e953-4307-bc38-88a6d2f47e50 None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:21.994073 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e65ba4-e953-4307-bc38-88a6d2f47e50 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:21.995484 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e65ba4-e953-4307-bc38-88a6d2f47e50 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:22.003198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 255/511] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:27:21 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:22.003948 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e65ba4-e953-4307-bc38-88a6d2f47e50 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:22.020756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:22.039749 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50efd386-f692-49b2-b595-f61d9b2b1881 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:22.073287 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63acddd-e06b-4ec2-84d1-29b62ba3ba8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:22.073894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63acddd-e06b-4ec2-84d1-29b62ba3ba8c None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:22.074179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63acddd-e06b-4ec2-84d1-29b62ba3ba8c None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:22.074459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63acddd-e06b-4ec2-84d1-29b62ba3ba8c None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:22.075211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63acddd-e06b-4ec2-84d1-29b62ba3ba8c None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:22.075523 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63acddd-e06b-4ec2-84d1-29b62ba3ba8c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:22.076969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63acddd-e06b-4ec2-84d1-29b62ba3ba8c None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:22.094338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 257/512] 10.222.0.39 () {56 vars in 1137 bytes} [Fri Sep 30 09:27:21 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:22.094996 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63acddd-e06b-4ec2-84d1-29b62ba3ba8c None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:22.100224 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e65ba4-e953-4307-bc38-88a6d2f47e50 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:22.120040 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:22.133834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a9e680-ff5f-4fff-b560-d257553d8fb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:22.134435 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a9e680-ff5f-4fff-b560-d257553d8fb1 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:22.134755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a9e680-ff5f-4fff-b560-d257553d8fb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:22.135033 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a9e680-ff5f-4fff-b560-d257553d8fb1 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:22.135675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a9e680-ff5f-4fff-b560-d257553d8fb1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:22.137190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a9e680-ff5f-4fff-b560-d257553d8fb1 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:22.196212 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63acddd-e06b-4ec2-84d1-29b62ba3ba8c None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:22.215036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 256/513] 10.222.0.39 () {60 vars in 1132 bytes} [Fri Sep 30 09:27:22 2022] POST /identity/v3/users => generated 247 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:27:22.215859 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a9e680-ff5f-4fff-b560-d257553d8fb1 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:22.242349 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba10ea2-66b6-4b5e-8a29-781b6354853a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:22.243461 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba10ea2-66b6-4b5e-8a29-781b6354853a None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:22.244041 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba10ea2-66b6-4b5e-8a29-781b6354853a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:22.244551 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba10ea2-66b6-4b5e-8a29-781b6354853a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:22.255263 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:22.325748 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a9e680-ff5f-4fff-b560-d257553d8fb1 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:22.327052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba10ea2-66b6-4b5e-8a29-781b6354853a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:22.328698 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba10ea2-66b6-4b5e-8a29-781b6354853a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:22.368358 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f5ea4b-5762-4b9e-9664-2fe47998e12d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:22.369011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f5ea4b-5762-4b9e-9664-2fe47998e12d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:22.369291 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f5ea4b-5762-4b9e-9664-2fe47998e12d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:22.369567 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f5ea4b-5762-4b9e-9664-2fe47998e12d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:22.420027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:22.439899 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 258/514] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:27:22.440757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba10ea2-66b6-4b5e-8a29-781b6354853a None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:22.450626 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f5ea4b-5762-4b9e-9664-2fe47998e12d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:22.452712 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f5ea4b-5762-4b9e-9664-2fe47998e12d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:22.481426 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:22.540386 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba10ea2-66b6-4b5e-8a29-781b6354853a None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:22.574088 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 257/515] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:27:22.574088 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f5ea4b-5762-4b9e-9664-2fe47998e12d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:22.578354 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b89eb6-6d09-4688-904c-ba7569cb3797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:22.579056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b89eb6-6d09-4688-904c-ba7569cb3797 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:22.579343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b89eb6-6d09-4688-904c-ba7569cb3797 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:22.579628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b89eb6-6d09-4688-904c-ba7569cb3797 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:22.591493 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b89eb6-6d09-4688-904c-ba7569cb3797 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:22.594342 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b89eb6-6d09-4688-904c-ba7569cb3797 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:22.612317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-f7b89eb6-6d09-4688-904c-ba7569cb3797 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:22.612317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:22.612317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:22.612317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:22.612317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:22.612317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:22.612317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:22.612317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:22.612317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:22.612317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:22.612317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:22.612317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:22.612317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:22.612317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:22.612317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:22.612317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:22.613283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:22.613283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:22.613283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:22.613283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:22.613283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:22.613283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:22.613283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:22.613283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:22.613283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:22.613283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:22.613283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:22.613283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:22.613283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:22.613283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:22.613283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:22.613283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:22.613283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:22.614165 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:22.614165 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:22.614165 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:22.615772 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 259/516] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:22.616711 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b89eb6-6d09-4688-904c-ba7569cb3797 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:22.662331 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:22.686934 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f5ea4b-5762-4b9e-9664-2fe47998e12d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:22.709615 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:22.719438 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b89eb6-6d09-4688-904c-ba7569cb3797 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:22.722202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449a513d-af74-4f8c-811e-6ab4dd99d194 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:22.723080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449a513d-af74-4f8c-811e-6ab4dd99d194 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:22.723526 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449a513d-af74-4f8c-811e-6ab4dd99d194 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:22.723801 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449a513d-af74-4f8c-811e-6ab4dd99d194 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:22.754363 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cad3b8b-1eee-4a36-898b-f43d30f853e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:22.755023 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cad3b8b-1eee-4a36-898b-f43d30f853e7 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:22.755292 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cad3b8b-1eee-4a36-898b-f43d30f853e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:22.755577 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cad3b8b-1eee-4a36-898b-f43d30f853e7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:22.767099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cad3b8b-1eee-4a36-898b-f43d30f853e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:22.768553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cad3b8b-1eee-4a36-898b-f43d30f853e7 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:22.784514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-0cad3b8b-1eee-4a36-898b-f43d30f853e7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:22.784514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:22.784514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:22.784514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:22.784514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:22.784514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:22.784514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:22.784514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:22.784514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:22.784514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:22.784514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:22.784514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:22.784514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:22.784514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:22.784514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:22.784514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:22.785528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:22.785528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:22.785528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:22.785528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:22.785528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:22.785528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:22.785528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:22.785528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:22.785528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:22.785528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:22.785528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:22.785528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:22.785528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:22.785528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:22.785528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:22.785528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:22.785528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:22.786422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:22.786422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:22.786422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:22.786422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 260/517] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:22.786422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cad3b8b-1eee-4a36-898b-f43d30f853e7 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:22.800630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449a513d-af74-4f8c-811e-6ab4dd99d194 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:22.802075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449a513d-af74-4f8c-811e-6ab4dd99d194 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:22.889974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:22.894453 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cad3b8b-1eee-4a36-898b-f43d30f853e7 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:22.917270 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 258/518] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:27:22.917270 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449a513d-af74-4f8c-811e-6ab4dd99d194 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:22.933467 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4fb1087-96fb-4bb4-a9dc-c6a77b144bc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:22.934101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fb1087-96fb-4bb4-a9dc-c6a77b144bc2 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:22.934378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fb1087-96fb-4bb4-a9dc-c6a77b144bc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:22.934680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fb1087-96fb-4bb4-a9dc-c6a77b144bc2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:22.946715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4fb1087-96fb-4bb4-a9dc-c6a77b144bc2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:22.948451 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4fb1087-96fb-4bb4-a9dc-c6a77b144bc2 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:22.950272 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:22.965653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-f4fb1087-96fb-4bb4-a9dc-c6a77b144bc2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:22.965653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:22.965653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:22.965653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:22.965653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:22.965653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:22.965653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:22.965653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:22.965653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:22.965653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:22.965653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:22.965653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:22.965653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:22.965653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:22.965653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:22.965653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:22.966820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:22.966820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:22.966820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:22.966820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:22.966820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:22.966820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:22.966820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:22.966820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:22.966820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:22.966820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:22.966820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:22.966820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:22.966820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:22.966820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:22.966820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:22.966820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:22.966820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:22.972281 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:22.972281 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:22.972281 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:22.972281 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 261/519] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:22.972281 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4fb1087-96fb-4bb4-a9dc-c6a77b144bc2 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:23.025700 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449a513d-af74-4f8c-811e-6ab4dd99d194 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:23.064590 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aaebce8-6858-4628-8749-fdfef1721625 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:23.065195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaebce8-6858-4628-8749-fdfef1721625 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:23.065476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaebce8-6858-4628-8749-fdfef1721625 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:23.065747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaebce8-6858-4628-8749-fdfef1721625 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:23.066379 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aaebce8-6858-4628-8749-fdfef1721625 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:23.066706 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aaebce8-6858-4628-8749-fdfef1721625 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:23.068092 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aaebce8-6858-4628-8749-fdfef1721625 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:23.078547 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4fb1087-96fb-4bb4-a9dc-c6a77b144bc2 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:23.079898 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 259/520] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:23.081640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aaebce8-6858-4628-8749-fdfef1721625 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:23.116618 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:23.151493 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5d0f89-7053-4a19-b44f-aee13b185710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:23.152225 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5d0f89-7053-4a19-b44f-aee13b185710 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:23.152508 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5d0f89-7053-4a19-b44f-aee13b185710 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:23.152779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5d0f89-7053-4a19-b44f-aee13b185710 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:23.153434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5d0f89-7053-4a19-b44f-aee13b185710 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:23.153734 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5d0f89-7053-4a19-b44f-aee13b185710 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:23.155218 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5d0f89-7053-4a19-b44f-aee13b185710 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:23.167886 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 262/521] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:23.168613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5d0f89-7053-4a19-b44f-aee13b185710 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:23.191703 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aaebce8-6858-4628-8749-fdfef1721625 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:23.229170 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391bed4f-46e7-476b-a0e9-a34c87c2feab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:23.230201 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:23.230658 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391bed4f-46e7-476b-a0e9-a34c87c2feab None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:23.231194 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391bed4f-46e7-476b-a0e9-a34c87c2feab None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:23.231423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391bed4f-46e7-476b-a0e9-a34c87c2feab None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:23.241987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391bed4f-46e7-476b-a0e9-a34c87c2feab None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:23.243474 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391bed4f-46e7-476b-a0e9-a34c87c2feab None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:23.259056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-391bed4f-46e7-476b-a0e9-a34c87c2feab None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:23.259056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:23.259056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:23.259056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:23.259056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:23.259056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:23.259056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:23.259056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:23.259056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:23.259056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:23.259056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:23.259056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:23.259056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:23.259056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:23.259056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:23.259056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:23.260159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:23.260159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:23.260159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:23.260159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:23.260159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:23.260159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:23.260159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:23.260159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:23.260159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:23.260159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:23.260159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:23.260159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:23.260159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:23.260159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:23.260159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:23.260159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:23.260159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:23.261038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:23.261038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:23.261038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:23.261038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 260/522] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:23.261665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391bed4f-46e7-476b-a0e9-a34c87c2feab None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:23.280159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5d0f89-7053-4a19-b44f-aee13b185710 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:23.316473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5774a3bc-b0ec-4efc-9d02-42d67f063336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:23.317239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774a3bc-b0ec-4efc-9d02-42d67f063336 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:23.317858 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774a3bc-b0ec-4efc-9d02-42d67f063336 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:23.317858 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774a3bc-b0ec-4efc-9d02-42d67f063336 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:23.318412 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5774a3bc-b0ec-4efc-9d02-42d67f063336 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:23.318743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5774a3bc-b0ec-4efc-9d02-42d67f063336 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:23.320126 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5774a3bc-b0ec-4efc-9d02-42d67f063336 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:23.336500 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 263/523] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:23.336500 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5774a3bc-b0ec-4efc-9d02-42d67f063336 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:23.358713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:23.371886 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391bed4f-46e7-476b-a0e9-a34c87c2feab None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:23.409798 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce80684a-8cfe-438e-8f08-af82eb5047e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:23.410792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80684a-8cfe-438e-8f08-af82eb5047e5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:23.411274 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80684a-8cfe-438e-8f08-af82eb5047e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:23.411863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80684a-8cfe-438e-8f08-af82eb5047e5 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:23.420419 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce80684a-8cfe-438e-8f08-af82eb5047e5 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:23.422120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce80684a-8cfe-438e-8f08-af82eb5047e5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:23.434430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-ce80684a-8cfe-438e-8f08-af82eb5047e5 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 30 09:27:23.434430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:23.434430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:23.434430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:23.434430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:23.434430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:23.434430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:23.434430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:23.434430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:23.434430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:23.434430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:23.434430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:23.434430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:23.434430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:23.434430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:23.434430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:23.436137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:23.436137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:23.436137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:23.436137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:23.436137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:23.436137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:23.436137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:23.436137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:23.436137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:23.436137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:23.436137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:23.436137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:23.436137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:23.436137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:23.436137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:23.436137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:23.436137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:23.436137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:23.437052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:23.437052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:23.437052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:23.437052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:23.437052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:23.437052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:23.437052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:23.437052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:23.437052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 30 09:27:23.437052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:23.437595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5774a3bc-b0ec-4efc-9d02-42d67f063336 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:23.440813 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 261/524] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:23 2022] GET /identity/v3/users/neutron => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:23.441435 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce80684a-8cfe-438e-8f08-af82eb5047e5 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:23.452310 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:23.479264 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e74febc-e75e-4fd8-bfb4-7cff942849e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:23.479829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e74febc-e75e-4fd8-bfb4-7cff942849e1 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:23.480109 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e74febc-e75e-4fd8-bfb4-7cff942849e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:23.480378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e74febc-e75e-4fd8-bfb4-7cff942849e1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:23.489250 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e74febc-e75e-4fd8-bfb4-7cff942849e1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:23.491938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e74febc-e75e-4fd8-bfb4-7cff942849e1 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:23.508368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-0e74febc-e75e-4fd8-bfb4-7cff942849e1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:23.508368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:23.508368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:23.508368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:23.508368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:23.508368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:23.508368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:23.508368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:23.508368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:23.508368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:23.508368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:23.508368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:23.508368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:23.508368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:23.508368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:23.509271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:23.509271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:23.509271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:23.509271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:23.509271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:23.509271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:23.509271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:23.509271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:23.509271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:23.509271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:23.509271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:23.509271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:23.509271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:23.509271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:23.509271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:23.509271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:23.509271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:23.510207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:23.510207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:23.510207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:23.510207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:23.510207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:23.510207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:23.510207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:23.510207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:23.510207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 264/525] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:23.510704 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e74febc-e75e-4fd8-bfb4-7cff942849e1 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:23.552870 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce80684a-8cfe-438e-8f08-af82eb5047e5 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:23.590825 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89429d03-17df-41b1-8ffb-d23eeeb7fb10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:23.592385 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89429d03-17df-41b1-8ffb-d23eeeb7fb10 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:23.592672 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89429d03-17df-41b1-8ffb-d23eeeb7fb10 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:23.592952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89429d03-17df-41b1-8ffb-d23eeeb7fb10 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:23.593998 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89429d03-17df-41b1-8ffb-d23eeeb7fb10 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:23.594291 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89429d03-17df-41b1-8ffb-d23eeeb7fb10 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:23.596341 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89429d03-17df-41b1-8ffb-d23eeeb7fb10 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:23.609450 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 262/526] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:23.614572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89429d03-17df-41b1-8ffb-d23eeeb7fb10 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:23.616743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e74febc-e75e-4fd8-bfb4-7cff942849e1 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:23.627564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:23.653427 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdf8f0e-ee00-4206-a39a-e421003a777d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:23.654553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdf8f0e-ee00-4206-a39a-e421003a777d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:23.654858 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdf8f0e-ee00-4206-a39a-e421003a777d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:23.655123 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdf8f0e-ee00-4206-a39a-e421003a777d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:23.661812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdf8f0e-ee00-4206-a39a-e421003a777d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:23.663286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdf8f0e-ee00-4206-a39a-e421003a777d None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:23.675005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-3fdf8f0e-ee00-4206-a39a-e421003a777d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:23.675005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:23.675005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:23.675005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:23.675005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:23.675005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:23.675005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:23.675005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:23.675005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:23.675005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:23.675005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:23.675005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:23.675005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:23.675005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:23.675005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:23.675880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:23.675880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:23.675880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:23.675880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:23.675880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:23.675880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:23.675880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:23.675880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:23.675880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:23.675880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:23.675880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:23.675880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:23.675880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:23.675880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:23.675880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:23.675880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:23.675880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:23.676739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:23.676739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:23.676739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:23.676739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:23.676739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:23.676739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:23.676739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:23.676739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:23.676739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 265/527] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:23.677374 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdf8f0e-ee00-4206-a39a-e421003a777d None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:23.715115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:23.735796 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89429d03-17df-41b1-8ffb-d23eeeb7fb10 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:23.774904 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6e376a-175b-47a6-9e00-63fd9947cdb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:23.775840 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6e376a-175b-47a6-9e00-63fd9947cdb1 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:23.776303 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6e376a-175b-47a6-9e00-63fd9947cdb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:23.776574 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6e376a-175b-47a6-9e00-63fd9947cdb1 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:23.777287 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6e376a-175b-47a6-9e00-63fd9947cdb1 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:23.777590 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6e376a-175b-47a6-9e00-63fd9947cdb1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:23.779080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6e376a-175b-47a6-9e00-63fd9947cdb1 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:23.786139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdf8f0e-ee00-4206-a39a-e421003a777d None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:23.795950 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 263/528] 10.222.0.39 () {56 vars in 1139 bytes} [Fri Sep 30 09:27:23 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:23.796754 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6e376a-175b-47a6-9e00-63fd9947cdb1 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:23.828639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cde1f5-6b94-4a25-958b-a8478fe1e0a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:23.829237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cde1f5-6b94-4a25-958b-a8478fe1e0a9 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:23.829516 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cde1f5-6b94-4a25-958b-a8478fe1e0a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:23.829792 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cde1f5-6b94-4a25-958b-a8478fe1e0a9 None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:23.830473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cde1f5-6b94-4a25-958b-a8478fe1e0a9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:23.830796 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cde1f5-6b94-4a25-958b-a8478fe1e0a9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:23.832235 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cde1f5-6b94-4a25-958b-a8478fe1e0a9 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:23.850006 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 266/529] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:23.850911 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cde1f5-6b94-4a25-958b-a8478fe1e0a9 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:23.874334 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:23.911167 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6e376a-175b-47a6-9e00-63fd9947cdb1 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:23.946809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:23.952969 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe0205a-0659-4517-9397-cafabdfdf831 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:23.953601 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe0205a-0659-4517-9397-cafabdfdf831 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:23.953735 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cde1f5-6b94-4a25-958b-a8478fe1e0a9 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:23.953886 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe0205a-0659-4517-9397-cafabdfdf831 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:23.954141 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe0205a-0659-4517-9397-cafabdfdf831 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:23.961024 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe0205a-0659-4517-9397-cafabdfdf831 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:23.962479 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe0205a-0659-4517-9397-cafabdfdf831 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:23.977585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-efe0205a-0659-4517-9397-cafabdfdf831 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:23.977585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:23.977585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:23.977585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:23.977585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:23.977585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:23.977585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:23.977585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:23.977585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:23.977585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:23.977585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:23.977585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:23.977585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:23.977585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:23.977585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:23.978476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:23.978476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:23.978476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:23.978476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:23.978476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:23.978476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:23.978476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:23.978476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:23.978476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:23.978476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:23.978476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:23.978476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:23.978476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:23.978476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:23.978476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:23.978476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:23.978476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:23.979390 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:23.979390 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:23.979390 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:23.979390 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:23.979390 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:23.979390 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:23.979390 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:23.979390 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:23.979390 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 264/530] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:23.979895 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe0205a-0659-4517-9397-cafabdfdf831 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:23.990108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2131713a-9504-4ed6-979c-65042ad55115 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:23.990680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2131713a-9504-4ed6-979c-65042ad55115 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:23.990960 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2131713a-9504-4ed6-979c-65042ad55115 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:23.991233 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2131713a-9504-4ed6-979c-65042ad55115 None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:23.991918 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2131713a-9504-4ed6-979c-65042ad55115 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:23.992218 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2131713a-9504-4ed6-979c-65042ad55115 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:23.993653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2131713a-9504-4ed6-979c-65042ad55115 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:24.010188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 267/531] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:24.010188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2131713a-9504-4ed6-979c-65042ad55115 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:24.091516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe0205a-0659-4517-9397-cafabdfdf831 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:24.116947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2131713a-9504-4ed6-979c-65042ad55115 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:24.119653 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:24.132605 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ca0d7a-d441-404b-95de-2273c79d4bbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:24.133206 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ca0d7a-d441-404b-95de-2273c79d4bbb None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:24.133639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ca0d7a-d441-404b-95de-2273c79d4bbb None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:24.133908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ca0d7a-d441-404b-95de-2273c79d4bbb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:24.151783 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27389536-3d32-4211-a556-6eb2d8d339f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:24.152587 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27389536-3d32-4211-a556-6eb2d8d339f6 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:24.152875 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27389536-3d32-4211-a556-6eb2d8d339f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:24.153150 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27389536-3d32-4211-a556-6eb2d8d339f6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:24.153868 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27389536-3d32-4211-a556-6eb2d8d339f6 None admin] RBAC: Adding query filter params (role.id=144dd855f74945548ea65db37acda1c1, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=bf907d4033064fb8a2707b1bbb0a180b) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:24.154785 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27389536-3d32-4211-a556-6eb2d8d339f6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:24.156252 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27389536-3d32-4211-a556-6eb2d8d339f6 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:24.170931 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 268/532] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:27:24 2022] GET /identity/v3/role_assignments?user.id=bf907d4033064fb8a2707b1bbb0a180b&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=144dd855f74945548ea65db37acda1c1 => generated 259 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:24.170931 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27389536-3d32-4211-a556-6eb2d8d339f6 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:24.198090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:24.240478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ca0d7a-d441-404b-95de-2273c79d4bbb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:24.242175 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ca0d7a-d441-404b-95de-2273c79d4bbb None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:24.319424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27389536-3d32-4211-a556-6eb2d8d339f6 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:24.373141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5694701-9f82-47bc-86d1-20282bf7428e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:24.373763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5694701-9f82-47bc-86d1-20282bf7428e None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:24.374045 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5694701-9f82-47bc-86d1-20282bf7428e None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:24.374320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5694701-9f82-47bc-86d1-20282bf7428e None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:24.375044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5694701-9f82-47bc-86d1-20282bf7428e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:24.375359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5694701-9f82-47bc-86d1-20282bf7428e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:24.376252 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 265/533] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:27:24.376831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5694701-9f82-47bc-86d1-20282bf7428e None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:24.377103 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ca0d7a-d441-404b-95de-2273c79d4bbb None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:24.393846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 269/534] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:24.394741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5694701-9f82-47bc-86d1-20282bf7428e None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:24.415446 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:24.490117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:24.495540 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ca0d7a-d441-404b-95de-2273c79d4bbb None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:24.514038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5694701-9f82-47bc-86d1-20282bf7428e None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:24.533185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a5da32-07a7-477a-81ef-294b5237e6af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:24.533790 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a5da32-07a7-477a-81ef-294b5237e6af None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:24.534069 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a5da32-07a7-477a-81ef-294b5237e6af None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:24.534342 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a5da32-07a7-477a-81ef-294b5237e6af None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:24.535087 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a5da32-07a7-477a-81ef-294b5237e6af None admin] RBAC: Adding query filter params (role.id=144dd855f74945548ea65db37acda1c1, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=7ff68fb87700423ab53c90520a6bd3b9) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:24.535398 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a5da32-07a7-477a-81ef-294b5237e6af None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:24.536821 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a5da32-07a7-477a-81ef-294b5237e6af None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:24.542452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 266/535] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:27:24 2022] GET /identity/v3/role_assignments?user.id=7ff68fb87700423ab53c90520a6bd3b9&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=144dd855f74945548ea65db37acda1c1 => generated 259 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:24.543348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a5da32-07a7-477a-81ef-294b5237e6af None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:24.556063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b87375-4c94-4f46-a441-e72617cfb405 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:24.556720 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b87375-4c94-4f46-a441-e72617cfb405 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:24.557000 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b87375-4c94-4f46-a441-e72617cfb405 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:24.557272 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b87375-4c94-4f46-a441-e72617cfb405 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:24.570514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b87375-4c94-4f46-a441-e72617cfb405 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:24.572082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b87375-4c94-4f46-a441-e72617cfb405 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:24.589299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-f8b87375-4c94-4f46-a441-e72617cfb405 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:24.589299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:24.589299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:24.589299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:24.589299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:24.589299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:24.589299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:24.589299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:24.589299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:24.589299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:24.589299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:24.589299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:24.589299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:24.589299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:24.589299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:24.589299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:24.590319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:24.590319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:24.590319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:24.590319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:24.590319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:24.590319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:24.590319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:24.590319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:24.590319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:24.590319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:24.590319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:24.590319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:24.590319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:24.590319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:24.590319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:24.590319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:24.590319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:24.591267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:24.591267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:24.591267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:24.595014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 270/536] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:24.596538 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a859f1-b3ae-4d8c-808e-ea7c9e795340 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:24.596835 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a859f1-b3ae-4d8c-808e-ea7c9e795340 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:24.597108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a859f1-b3ae-4d8c-808e-ea7c9e795340 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:24.599851 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 271/537] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:24.600711 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a859f1-b3ae-4d8c-808e-ea7c9e795340 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:24.679440 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:24.694053 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a5da32-07a7-477a-81ef-294b5237e6af None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:24.734521 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a859f1-b3ae-4d8c-808e-ea7c9e795340 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:24.735884 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b78632-edc2-451c-bb64-32c4304ebc5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:24.736553 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b78632-edc2-451c-bb64-32c4304ebc5b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:24.736834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b78632-edc2-451c-bb64-32c4304ebc5b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:24.737104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b78632-edc2-451c-bb64-32c4304ebc5b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:24.737819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b78632-edc2-451c-bb64-32c4304ebc5b None admin] RBAC: Adding query filter params (role.id=144dd855f74945548ea65db37acda1c1, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=4e2742f567444dff88ed919be52217de) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:24.738123 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b78632-edc2-451c-bb64-32c4304ebc5b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:24.747158 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b78632-edc2-451c-bb64-32c4304ebc5b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:24.751102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 267/538] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:27:24 2022] GET /identity/v3/role_assignments?user.id=4e2742f567444dff88ed919be52217de&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=144dd855f74945548ea65db37acda1c1 => generated 259 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:24.753023 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56477d39-0ced-4feb-b718-ab88343e5144 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:24.753594 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56477d39-0ced-4feb-b718-ab88343e5144 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:24.753871 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56477d39-0ced-4feb-b718-ab88343e5144 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:24.774785 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031701a5-7a75-4c7b-b253-a34249fcc0e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:24.775600 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031701a5-7a75-4c7b-b253-a34249fcc0e9 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:24.776050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031701a5-7a75-4c7b-b253-a34249fcc0e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:24.776661 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031701a5-7a75-4c7b-b253-a34249fcc0e9 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:24.777452 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031701a5-7a75-4c7b-b253-a34249fcc0e9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:24.777837 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031701a5-7a75-4c7b-b253-a34249fcc0e9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:24.780464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031701a5-7a75-4c7b-b253-a34249fcc0e9 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:24.788314 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:24.794035 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 272/539] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:24.795779 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-56477d39-0ced-4feb-b718-ab88343e5144 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:24.797016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-56477d39-0ced-4feb-b718-ab88343e5144 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:24.799405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031701a5-7a75-4c7b-b253-a34249fcc0e9 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:24.927809 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 268/540] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:24.967568 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031701a5-7a75-4c7b-b253-a34249fcc0e9 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:25.004835 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6154ad06-61c3-4378-8a4f-107c5e5a6695 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:25.005799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6154ad06-61c3-4378-8a4f-107c5e5a6695 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:25.006282 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6154ad06-61c3-4378-8a4f-107c5e5a6695 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:25.007048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6154ad06-61c3-4378-8a4f-107c5e5a6695 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:25.015103 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6154ad06-61c3-4378-8a4f-107c5e5a6695 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:25.016800 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6154ad06-61c3-4378-8a4f-107c5e5a6695 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:25.028947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-6154ad06-61c3-4378-8a4f-107c5e5a6695 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:25.028947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:25.028947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:25.028947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:25.028947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:25.028947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:25.028947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:25.028947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:25.028947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:25.028947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:25.028947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:25.028947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:25.028947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:25.028947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:25.028947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:25.029872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:25.029872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:25.029872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:25.029872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:25.029872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:25.029872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:25.029872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:25.029872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:25.029872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:25.029872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:25.029872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:25.029872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:25.029872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:25.029872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:25.029872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:25.029872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:25.029872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:25.030782 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:25.030782 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:25.030782 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:25.030782 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:25.030782 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:25.030782 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:25.030782 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:25.030782 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:25.033170 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 273/541] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:25.038416 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56477d39-0ced-4feb-b718-ab88343e5144 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:25.148383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:25.165592 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56477d39-0ced-4feb-b718-ab88343e5144 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:25.209601 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dff35d0-fc62-4b14-82e7-6d94b74fa8c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:25.210341 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dff35d0-fc62-4b14-82e7-6d94b74fa8c9 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:25.210987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dff35d0-fc62-4b14-82e7-6d94b74fa8c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:25.211357 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dff35d0-fc62-4b14-82e7-6d94b74fa8c9 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:25.212156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dff35d0-fc62-4b14-82e7-6d94b74fa8c9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:25.212593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dff35d0-fc62-4b14-82e7-6d94b74fa8c9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:25.214193 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dff35d0-fc62-4b14-82e7-6d94b74fa8c9 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:25.238106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 269/542] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:25.243778 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6154ad06-61c3-4378-8a4f-107c5e5a6695 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:25.296382 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:25.343560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dff35d0-fc62-4b14-82e7-6d94b74fa8c9 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:25.354170 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6154ad06-61c3-4378-8a4f-107c5e5a6695 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:25.417653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c082ed94-a123-4a75-8edf-50ae6ae03114 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:25.418257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c082ed94-a123-4a75-8edf-50ae6ae03114 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:25.418540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c082ed94-a123-4a75-8edf-50ae6ae03114 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:25.418833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c082ed94-a123-4a75-8edf-50ae6ae03114 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:25.420086 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c082ed94-a123-4a75-8edf-50ae6ae03114 None admin] RBAC: Adding query filter params (role.id=144dd855f74945548ea65db37acda1c1, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=9266babd21414ae9998716e5a0478e9d) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:25.420394 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c082ed94-a123-4a75-8edf-50ae6ae03114 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:25.421840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c082ed94-a123-4a75-8edf-50ae6ae03114 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:25.438561 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 274/543] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:27:25 2022] GET /identity/v3/role_assignments?user.id=9266babd21414ae9998716e5a0478e9d&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=144dd855f74945548ea65db37acda1c1 => generated 259 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:25.500553 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dff35d0-fc62-4b14-82e7-6d94b74fa8c9 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:25.546684 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa54739-efc0-46a6-8b1d-a5380333a34d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:25.547264 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa54739-efc0-46a6-8b1d-a5380333a34d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:25.547548 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa54739-efc0-46a6-8b1d-a5380333a34d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:25.547880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa54739-efc0-46a6-8b1d-a5380333a34d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:25.565236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa54739-efc0-46a6-8b1d-a5380333a34d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:25.570408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-1fa54739-efc0-46a6-8b1d-a5380333a34d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:25.570408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:25.570408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:25.570408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:25.570408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:25.570408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:25.570408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:25.570408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:25.570408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:25.570408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:25.570408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:25.570408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:25.570408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:25.570408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:25.570408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:25.570408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:25.571424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:25.571424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:25.571424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:25.571424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:25.571424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:25.571424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:25.571424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:25.571424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:25.571424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:25.571424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:25.571424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:25.571424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:25.571424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:25.571424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:25.571424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:25.571424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:25.572241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:25.572241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:25.572241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:25.572241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:25.572241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:25.572241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:25.572241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:25.572241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:25.572241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:25.572241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:25.572241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:25.572241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:25.577357 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 270/544] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:25 2022] GET /identity/v3/roles/service => generated 85 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:25.579886 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c082ed94-a123-4a75-8edf-50ae6ae03114 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:25.721242 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c082ed94-a123-4a75-8edf-50ae6ae03114 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:25.742328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:25.771383 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c02e71-f25c-4154-a5cf-b1563783a6c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:25.773406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c02e71-f25c-4154-a5cf-b1563783a6c7 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:25.773406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c02e71-f25c-4154-a5cf-b1563783a6c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:25.773406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c02e71-f25c-4154-a5cf-b1563783a6c7 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:25.773406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c02e71-f25c-4154-a5cf-b1563783a6c7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:25.774005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c02e71-f25c-4154-a5cf-b1563783a6c7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:25.775351 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c02e71-f25c-4154-a5cf-b1563783a6c7 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:25.787682 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 275/545] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:27:25 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:25.793791 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa54739-efc0-46a6-8b1d-a5380333a34d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:25.798899 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:25.932488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa54739-efc0-46a6-8b1d-a5380333a34d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:25.969156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a3d709-cebc-498b-9447-f423794dbd02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:25.969794 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a3d709-cebc-498b-9447-f423794dbd02 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:25.970074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a3d709-cebc-498b-9447-f423794dbd02 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:25.970347 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a3d709-cebc-498b-9447-f423794dbd02 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:25.980834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a3d709-cebc-498b-9447-f423794dbd02 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:25.982907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a3d709-cebc-498b-9447-f423794dbd02 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:25.997787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-10a3d709-cebc-498b-9447-f423794dbd02 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:25.997787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:25.997787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:25.997787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:25.997787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:25.997787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:25.997787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:25.997787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:25.997787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:25.997787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:25.997787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:25.997787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:25.997787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:25.997787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:25.997787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:25.997787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:25.999945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:25.999945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:25.999945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:25.999945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:25.999945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:25.999945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:25.999945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:25.999945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:25.999945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:25.999945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:25.999945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:25.999945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:25.999945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:25.999945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:25.999945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:25.999945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:25.999945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:26.002104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:26.002104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:26.002104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:26.006799 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 271/546] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:26.011706 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c02e71-f25c-4154-a5cf-b1563783a6c7 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:26.164372 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c02e71-f25c-4154-a5cf-b1563783a6c7 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:26.209906 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e238e4-96be-4521-a636-103dc02cc322 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:26.210515 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e238e4-96be-4521-a636-103dc02cc322 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:26.210821 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e238e4-96be-4521-a636-103dc02cc322 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:26.211098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e238e4-96be-4521-a636-103dc02cc322 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:26.211735 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e238e4-96be-4521-a636-103dc02cc322 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:26.212033 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e238e4-96be-4521-a636-103dc02cc322 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:26.213458 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e238e4-96be-4521-a636-103dc02cc322 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:26.231592 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 276/547] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:26.237394 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a3d709-cebc-498b-9447-f423794dbd02 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:26.334139 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:26.391036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a3d709-cebc-498b-9447-f423794dbd02 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:26.452776 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab3972e-3b0b-45a0-97ad-c304aaa13361 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:26.453773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab3972e-3b0b-45a0-97ad-c304aaa13361 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:26.454292 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab3972e-3b0b-45a0-97ad-c304aaa13361 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:26.454801 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab3972e-3b0b-45a0-97ad-c304aaa13361 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:26.468419 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab3972e-3b0b-45a0-97ad-c304aaa13361 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:26.470171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab3972e-3b0b-45a0-97ad-c304aaa13361 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:26.487702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-8ab3972e-3b0b-45a0-97ad-c304aaa13361 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 30 09:27:26.487702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:26.487702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:26.487702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:26.487702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:26.487702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:26.487702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:26.487702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:26.487702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:26.487702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:26.487702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:26.487702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:26.487702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:26.487702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:26.487702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:26.487702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:26.488814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:26.488814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:26.488814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:26.488814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:26.488814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:26.488814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:26.488814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:26.488814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:26.488814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:26.488814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:26.488814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:26.488814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:26.488814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:26.488814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:26.488814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:26.488814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:26.488814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:26.488814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:26.489761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:26.489761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:26.489761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:26.489761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:26.489761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:26.489761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:26.489761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:26.489761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:26.489761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 30 09:27:26.489761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:26.492289 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 272/548] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:27:26 2022] GET /identity/v3/users/swift => generated 83 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:26.499667 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e238e4-96be-4521-a636-103dc02cc322 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:26.558960 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:26.639408 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e238e4-96be-4521-a636-103dc02cc322 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:26.689603 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261c5c5e-99ee-4718-8725-f7bdadebe4d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:26.690658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261c5c5e-99ee-4718-8725-f7bdadebe4d0 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:26.691147 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261c5c5e-99ee-4718-8725-f7bdadebe4d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:26.691634 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261c5c5e-99ee-4718-8725-f7bdadebe4d0 None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:26.692566 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261c5c5e-99ee-4718-8725-f7bdadebe4d0 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:26.693101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261c5c5e-99ee-4718-8725-f7bdadebe4d0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:26.694777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261c5c5e-99ee-4718-8725-f7bdadebe4d0 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:26.719529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 277/549] 10.222.0.39 () {56 vars in 1135 bytes} [Fri Sep 30 09:27:26 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:26.725270 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab3972e-3b0b-45a0-97ad-c304aaa13361 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:26.860014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab3972e-3b0b-45a0-97ad-c304aaa13361 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:26.886582 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:26.912459 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5946989-f3f0-4659-8676-ab283fc5f413 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:26.913100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5946989-f3f0-4659-8676-ab283fc5f413 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:26.913378 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5946989-f3f0-4659-8676-ab283fc5f413 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:26.913659 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5946989-f3f0-4659-8676-ab283fc5f413 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:26.967740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a594d76-0f96-4017-962e-dde89bc1e7b5 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:26.968115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a594d76-0f96-4017-962e-dde89bc1e7b5 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:26.968304 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a594d76-0f96-4017-962e-dde89bc1e7b5 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:26.969950 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 278/550] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:26.975051 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff43cd73-49ea-40d8-87f7-01d3edbfd8b4 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:26.975341 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff43cd73-49ea-40d8-87f7-01d3edbfd8b4 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:26.975619 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff43cd73-49ea-40d8-87f7-01d3edbfd8b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:26.994453 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-ff43cd73-49ea-40d8-87f7-01d3edbfd8b4 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:26.995759 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ff43cd73-49ea-40d8-87f7-01d3edbfd8b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:27.015095 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5946989-f3f0-4659-8676-ab283fc5f413 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:27.016821 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5946989-f3f0-4659-8676-ab283fc5f413 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:27.105603 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 279/551] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:27.108143 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82b9317-d6fc-4867-8207-8409381c9e5c None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:27.108664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82b9317-d6fc-4867-8207-8409381c9e5c None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:27.109170 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82b9317-d6fc-4867-8207-8409381c9e5c None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:27.110886 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 280/552] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:27.120452 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e3e3a9-628f-4af0-b644-93719250ef26 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:27.120967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e3e3a9-628f-4af0-b644-93719250ef26 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:27.121442 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e3e3a9-628f-4af0-b644-93719250ef26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:27.145592 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-32e3e3a9-628f-4af0-b644-93719250ef26 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:27.147165 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-32e3e3a9-628f-4af0-b644-93719250ef26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:27.189215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:27.279259 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 273/553] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 554 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:27:27.279582 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 281/554] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:27:27.283786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5946989-f3f0-4659-8676-ab283fc5f413 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:27.303192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:27.349819 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644a3cbb-7899-40dc-97a7-105389e977b8 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:27.350178 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644a3cbb-7899-40dc-97a7-105389e977b8 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:27.350377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644a3cbb-7899-40dc-97a7-105389e977b8 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:27.354621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 282/555] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:27.363207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889dbc8a-903b-4dc6-8b9d-37c6a85dab94 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:27.363487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889dbc8a-903b-4dc6-8b9d-37c6a85dab94 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:27.363760 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889dbc8a-903b-4dc6-8b9d-37c6a85dab94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:27.395342 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5946989-f3f0-4659-8676-ab283fc5f413 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:27.407272 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-889dbc8a-903b-4dc6-8b9d-37c6a85dab94 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:27.408455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-889dbc8a-903b-4dc6-8b9d-37c6a85dab94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:27.430910 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3f7745-34eb-4cb3-9010-998ec0d7259d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:27.431542 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3f7745-34eb-4cb3-9010-998ec0d7259d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:27.431823 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3f7745-34eb-4cb3-9010-998ec0d7259d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:27.432096 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3f7745-34eb-4cb3-9010-998ec0d7259d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:27.442687 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3f7745-34eb-4cb3-9010-998ec0d7259d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:27.444165 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3f7745-34eb-4cb3-9010-998ec0d7259d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:27.459786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-1f3f7745-34eb-4cb3-9010-998ec0d7259d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:27.459786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:27.459786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:27.459786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:27.459786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:27.459786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:27.459786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:27.459786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:27.459786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:27.459786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:27.459786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:27.459786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:27.459786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:27.459786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:27.459786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:27.459786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:27.460734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:27.460734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:27.460734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:27.460734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:27.460734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:27.460734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:27.460734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:27.460734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:27.460734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:27.460734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:27.460734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:27.460734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:27.460734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:27.460734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:27.460734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:27.460734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:27.460734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:27.461635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:27.461635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:27.461635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:27.461635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 274/556] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:27.462546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3f7745-34eb-4cb3-9010-998ec0d7259d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:27.509254 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 283/557] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:27.510432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889dbc8a-903b-4dc6-8b9d-37c6a85dab94 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:27.538411 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:27.574436 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3f7745-34eb-4cb3-9010-998ec0d7259d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:27.615670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2e6f0a-f3e4-4ac7-b07a-73f6b003c715 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:27.616302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2e6f0a-f3e4-4ac7-b07a-73f6b003c715 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:27.616882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2e6f0a-f3e4-4ac7-b07a-73f6b003c715 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:27.617155 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2e6f0a-f3e4-4ac7-b07a-73f6b003c715 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:27.629526 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2e6f0a-f3e4-4ac7-b07a-73f6b003c715 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:27.636462 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-4f2e6f0a-f3e4-4ac7-b07a-73f6b003c715 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:27.636462 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:27.636462 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:27.636462 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:27.636462 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:27.636462 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:27.636462 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:27.636462 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:27.636462 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:27.636462 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:27.636462 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:27.636462 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:27.636462 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:27.636462 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:27.636462 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:27.636462 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:27.637564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:27.637564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:27.637564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:27.637564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:27.637564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:27.637564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:27.637564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:27.637564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:27.637564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:27.637564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:27.637564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:27.637564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:27.637564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:27.637564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:27.637564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:27.637564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:27.638982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:27.638982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:27.638982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:27.638982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:27.638982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:27.638982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:27.638982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:27.638982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:27.638982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:27.638982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:27.638982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:27.638982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:27.638982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 275/558] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:27.638982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2e6f0a-f3e4-4ac7-b07a-73f6b003c715 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:27.646838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:27.662182 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889dbc8a-903b-4dc6-8b9d-37c6a85dab94 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:27.704681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780b0057-4a36-47f1-ac1e-7fac4f83e28d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:27.705796 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780b0057-4a36-47f1-ac1e-7fac4f83e28d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:27.706430 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780b0057-4a36-47f1-ac1e-7fac4f83e28d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:27.706750 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780b0057-4a36-47f1-ac1e-7fac4f83e28d None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:27.707660 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-780b0057-4a36-47f1-ac1e-7fac4f83e28d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:27.707977 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-780b0057-4a36-47f1-ac1e-7fac4f83e28d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:27.710057 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-780b0057-4a36-47f1-ac1e-7fac4f83e28d None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:27.726248 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 284/559] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:27.727597 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780b0057-4a36-47f1-ac1e-7fac4f83e28d None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:27.753539 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2e6f0a-f3e4-4ac7-b07a-73f6b003c715 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:27.789330 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:27.797848 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f301fd2f-02d9-45ff-9a96-4ecea96d25ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:27.802293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f301fd2f-02d9-45ff-9a96-4ecea96d25ab None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:27.802293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f301fd2f-02d9-45ff-9a96-4ecea96d25ab None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:27.802293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f301fd2f-02d9-45ff-9a96-4ecea96d25ab None admin] PATH_INFO: `/v3/roles/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:27.815855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f301fd2f-02d9-45ff-9a96-4ecea96d25ab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:27.820769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-f301fd2f-02d9-45ff-9a96-4ecea96d25ab None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:27.820769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:27.820769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:27.820769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:27.820769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:27.820769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:27.820769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:27.820769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:27.820769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:27.820769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:27.820769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:27.820769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:27.820769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:27.820769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:27.820769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:27.820769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:27.821900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:27.821900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:27.821900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:27.821900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:27.821900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:27.821900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:27.821900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:27.821900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:27.821900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:27.821900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:27.821900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:27.821900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:27.821900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:27.821900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:27.821900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:27.821900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:27.822974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:27.822974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:27.822974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:27.822974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:27.822974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:27.822974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:27.822974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:27.822974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:27.822974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:27.822974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:27.822974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:27.822974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:27.830581 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 276/560] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:27.831553 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f301fd2f-02d9-45ff-9a96-4ecea96d25ab None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:27.866473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780b0057-4a36-47f1-ac1e-7fac4f83e28d None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:27.907606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9f556d-ddcf-4e43-811a-d2a171bb95dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:27.908207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9f556d-ddcf-4e43-811a-d2a171bb95dd None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:27.908482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9f556d-ddcf-4e43-811a-d2a171bb95dd None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:27.908750 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9f556d-ddcf-4e43-811a-d2a171bb95dd None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:27.909398 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9f556d-ddcf-4e43-811a-d2a171bb95dd None admin] RBAC: Adding query filter params (name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:27.909694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9f556d-ddcf-4e43-811a-d2a171bb95dd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:27.911087 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9f556d-ddcf-4e43-811a-d2a171bb95dd None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:27.923718 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 285/561] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:27:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:27.923946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9f556d-ddcf-4e43-811a-d2a171bb95dd None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:27.936018 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:27.939653 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f301fd2f-02d9-45ff-9a96-4ecea96d25ab None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:27.975810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97830476-c46c-49ec-a8fd-aed793c80e69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:27.976454 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97830476-c46c-49ec-a8fd-aed793c80e69 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:27.976736 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97830476-c46c-49ec-a8fd-aed793c80e69 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:27.977006 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97830476-c46c-49ec-a8fd-aed793c80e69 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:27.983733 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97830476-c46c-49ec-a8fd-aed793c80e69 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:27.985198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97830476-c46c-49ec-a8fd-aed793c80e69 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:27.996555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-97830476-c46c-49ec-a8fd-aed793c80e69 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:27.996555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:27.996555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:27.996555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:27.996555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:27.996555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:27.996555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:27.996555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:27.996555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:27.996555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:27.996555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:27.996555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:27.996555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:27.996555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:27.996555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:27.998002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:27.998002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:27.998002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:27.998002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:27.998002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:27.998002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:27.998002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:27.998002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:27.998002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:27.998002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:27.998002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:27.998002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:27.998002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:27.998002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:27.998002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:27.998002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:27.998002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:28.003911 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:28.003911 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:28.003911 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:28.003911 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:28.003911 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:28.003911 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:28.003911 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:28.003911 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:28.003911 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 277/562] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:28.003911 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97830476-c46c-49ec-a8fd-aed793c80e69 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:28.037203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9f556d-ddcf-4e43-811a-d2a171bb95dd None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:28.053661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:28.074195 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc255967-ea93-4d9a-89a5-3486ef8d908f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:28.075384 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc255967-ea93-4d9a-89a5-3486ef8d908f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:28.075384 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc255967-ea93-4d9a-89a5-3486ef8d908f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:28.075384 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc255967-ea93-4d9a-89a5-3486ef8d908f None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:28.076566 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc255967-ea93-4d9a-89a5-3486ef8d908f None admin] RBAC: Adding query filter params (name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:28.076869 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc255967-ea93-4d9a-89a5-3486ef8d908f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:28.078258 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc255967-ea93-4d9a-89a5-3486ef8d908f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:28.091519 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 286/563] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:27:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:28.092238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc255967-ea93-4d9a-89a5-3486ef8d908f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:28.110366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97830476-c46c-49ec-a8fd-aed793c80e69 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:28.149956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dacbfe9-204e-495e-ab4a-2f4975246e0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:28.150668 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dacbfe9-204e-495e-ab4a-2f4975246e0f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:28.150912 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dacbfe9-204e-495e-ab4a-2f4975246e0f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:28.151191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dacbfe9-204e-495e-ab4a-2f4975246e0f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:28.163575 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dacbfe9-204e-495e-ab4a-2f4975246e0f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:28.163981 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:28.169732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-5dacbfe9-204e-495e-ab4a-2f4975246e0f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:28.169732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:28.169732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:28.169732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:28.169732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:28.169732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:28.169732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:28.169732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:28.169732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:28.169732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:28.169732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:28.169732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:28.169732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:28.169732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:28.169732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:28.169732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:28.170966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:28.170966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:28.170966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:28.170966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:28.170966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:28.170966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:28.170966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:28.170966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:28.170966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:28.170966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:28.170966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:28.170966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:28.170966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:28.170966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:28.170966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:28.170966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:28.172312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:28.172312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:28.172312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:28.172312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:28.172312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:28.172312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:28.172312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:28.172312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:28.172312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:28.172312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:28.172312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:28.172312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:28.175920 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 278/564] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:28.179368 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dacbfe9-204e-495e-ab4a-2f4975246e0f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:28.205588 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc255967-ea93-4d9a-89a5-3486ef8d908f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:28.239559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b3e844-c926-4e64-be9d-beff917caece None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:28.240416 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b3e844-c926-4e64-be9d-beff917caece None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:28.240854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b3e844-c926-4e64-be9d-beff917caece None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:28.241271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b3e844-c926-4e64-be9d-beff917caece None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:28.253546 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b3e844-c926-4e64-be9d-beff917caece None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:28.255042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b3e844-c926-4e64-be9d-beff917caece None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:28.269941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-e7b3e844-c926-4e64-be9d-beff917caece None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:28.269941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:28.269941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:28.269941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:28.269941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:28.269941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:28.269941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:28.269941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:28.269941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:28.269941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:28.269941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:28.269941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:28.269941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:28.269941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:28.269941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:28.269941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:28.270991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:28.270991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:28.270991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:28.270991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:28.270991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:28.270991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:28.270991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:28.270991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:28.270991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:28.270991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:28.270991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:28.270991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:28.270991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:28.270991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:28.270991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:28.270991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:28.270991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:28.271904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:28.271904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:28.271904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:28.271904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 287/565] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:28.271904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b3e844-c926-4e64-be9d-beff917caece None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:28.290028 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dacbfe9-204e-495e-ab4a-2f4975246e0f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:28.298658 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:28.323461 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e4eb29-babf-4ff0-972e-6e9e85fbd518 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:28.323992 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e4eb29-babf-4ff0-972e-6e9e85fbd518 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:28.324266 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e4eb29-babf-4ff0-972e-6e9e85fbd518 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:28.324534 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e4eb29-babf-4ff0-972e-6e9e85fbd518 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:28.325207 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e4eb29-babf-4ff0-972e-6e9e85fbd518 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:28.326046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e4eb29-babf-4ff0-972e-6e9e85fbd518 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:28.327546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e4eb29-babf-4ff0-972e-6e9e85fbd518 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:28.339911 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 279/566] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:28.340687 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e4eb29-babf-4ff0-972e-6e9e85fbd518 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:28.370756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b3e844-c926-4e64-be9d-beff917caece None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:28.390723 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:28.404214 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29399e25-c545-4fbe-9cf0-52807d308de6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:28.404842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29399e25-c545-4fbe-9cf0-52807d308de6 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:28.405123 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29399e25-c545-4fbe-9cf0-52807d308de6 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:28.405399 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29399e25-c545-4fbe-9cf0-52807d308de6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:28.415697 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29399e25-c545-4fbe-9cf0-52807d308de6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:28.417310 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29399e25-c545-4fbe-9cf0-52807d308de6 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:28.432465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-29399e25-c545-4fbe-9cf0-52807d308de6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:28.432465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:28.432465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:28.432465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:28.432465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:28.432465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:28.432465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:28.432465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:28.432465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:28.432465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:28.432465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:28.432465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:28.432465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:28.432465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:28.432465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:28.432465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:28.433598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:28.433598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:28.433598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:28.433598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:28.433598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:28.433598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:28.433598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:28.433598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:28.433598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:28.433598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:28.433598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:28.433598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:28.433598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:28.433598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:28.433598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:28.433598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:28.433598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:28.434507 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:28.434507 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:28.434507 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:28.434507 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 288/567] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:28.434790 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4153b1cf-cff3-4e05-96a0-05552ac09f88 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:28.435642 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4153b1cf-cff3-4e05-96a0-05552ac09f88 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:28.435812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4153b1cf-cff3-4e05-96a0-05552ac09f88 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:28.436975 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 289/568] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:28.438932 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e4eb29-babf-4ff0-972e-6e9e85fbd518 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:28.439612 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4153b1cf-cff3-4e05-96a0-05552ac09f88 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:28.471915 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6bded3-ebd1-4b28-80ab-f615db571962 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:28.472510 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6bded3-ebd1-4b28-80ab-f615db571962 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:28.472797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6bded3-ebd1-4b28-80ab-f615db571962 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:28.473061 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6bded3-ebd1-4b28-80ab-f615db571962 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:28.473709 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6bded3-ebd1-4b28-80ab-f615db571962 None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:28.474172 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6bded3-ebd1-4b28-80ab-f615db571962 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:28.476269 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6bded3-ebd1-4b28-80ab-f615db571962 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:28.485583 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 280/569] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:27:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:28.486341 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6bded3-ebd1-4b28-80ab-f615db571962 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:28.515153 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:28.540814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4153b1cf-cff3-4e05-96a0-05552ac09f88 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:28.575053 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec086d93-6716-46a3-a4be-7f02672979c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:28.575583 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec086d93-6716-46a3-a4be-7f02672979c9 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:28.575859 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec086d93-6716-46a3-a4be-7f02672979c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:28.576129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec086d93-6716-46a3-a4be-7f02672979c9 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:28.576755 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec086d93-6716-46a3-a4be-7f02672979c9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:28.577050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec086d93-6716-46a3-a4be-7f02672979c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:28.578412 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec086d93-6716-46a3-a4be-7f02672979c9 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:28.581745 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6bded3-ebd1-4b28-80ab-f615db571962 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:28.589133 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 290/570] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:28.589912 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec086d93-6716-46a3-a4be-7f02672979c9 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:28.614280 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdf7d7c-528b-4e0c-aed5-2f049146e641 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:28.614905 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdf7d7c-528b-4e0c-aed5-2f049146e641 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:28.615184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdf7d7c-528b-4e0c-aed5-2f049146e641 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:28.615455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdf7d7c-528b-4e0c-aed5-2f049146e641 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:28.616162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afdf7d7c-528b-4e0c-aed5-2f049146e641 None admin] RBAC: Adding query filter params (role.id=144dd855f74945548ea65db37acda1c1, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=0e5ad921e6fa41c297696ec7279f0fc4) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:28.616469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afdf7d7c-528b-4e0c-aed5-2f049146e641 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:28.617882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afdf7d7c-528b-4e0c-aed5-2f049146e641 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:28.621779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:28.624006 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 281/571] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:27:28 2022] GET /identity/v3/role_assignments?user.id=0e5ad921e6fa41c297696ec7279f0fc4&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=144dd855f74945548ea65db37acda1c1 => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:28.625117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c930a972-5156-46ab-9e40-cd4f6f4a6759 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:28.625409 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c930a972-5156-46ab-9e40-cd4f6f4a6759 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:28.625680 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c930a972-5156-46ab-9e40-cd4f6f4a6759 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:28.655730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c930a972-5156-46ab-9e40-cd4f6f4a6759 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:28.656978 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c930a972-5156-46ab-9e40-cd4f6f4a6759 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:28.702612 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec086d93-6716-46a3-a4be-7f02672979c9 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:28.735167 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 282/572] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:28.735667 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c930a972-5156-46ab-9e40-cd4f6f4a6759 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:28.743952 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0f7a8c-84f9-4e4f-84c6-0a2179c49b73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:28.744963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0f7a8c-84f9-4e4f-84c6-0a2179c49b73 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:28.745452 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0f7a8c-84f9-4e4f-84c6-0a2179c49b73 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:28.745983 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0f7a8c-84f9-4e4f-84c6-0a2179c49b73 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:28.746877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0f7a8c-84f9-4e4f-84c6-0a2179c49b73 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:28.747434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0f7a8c-84f9-4e4f-84c6-0a2179c49b73 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:28.749582 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0f7a8c-84f9-4e4f-84c6-0a2179c49b73 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:28.768002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:28.769171 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 291/573] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:28.770407 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0f7a8c-84f9-4e4f-84c6-0a2179c49b73 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:28.863623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c930a972-5156-46ab-9e40-cd4f6f4a6759 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:28.889392 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:28.901252 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0f7a8c-84f9-4e4f-84c6-0a2179c49b73 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:28.904279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9ef5b5-2bb8-415f-8f6b-8f062a2b931a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:28.905712 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9ef5b5-2bb8-415f-8f6b-8f062a2b931a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:28.905993 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9ef5b5-2bb8-415f-8f6b-8f062a2b931a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:28.906266 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9ef5b5-2bb8-415f-8f6b-8f062a2b931a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:28.918997 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9ef5b5-2bb8-415f-8f6b-8f062a2b931a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:28.921313 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9ef5b5-2bb8-415f-8f6b-8f062a2b931a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:28.939952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-bb9ef5b5-2bb8-415f-8f6b-8f062a2b931a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:28.939952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:28.939952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:28.939952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:28.939952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:28.939952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:28.939952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:28.939952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:28.939952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:28.939952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:28.939952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:28.939952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:28.939952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:28.939952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:28.939952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:28.939952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:28.941635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:28.941635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:28.941635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:28.941635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:28.941635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:28.941635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:28.941635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:28.941635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:28.941635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:28.941635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:28.941635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:28.941635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:28.941635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:28.941635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:28.941635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:28.941635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:28.941635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:28.942738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:28.942738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:28.942738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:28.942738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 283/574] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:28.942738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9ef5b5-2bb8-415f-8f6b-8f062a2b931a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:28.949969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710d69b1-1cb5-42fe-b05d-68fac8375746 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:28.950895 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710d69b1-1cb5-42fe-b05d-68fac8375746 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:28.951324 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710d69b1-1cb5-42fe-b05d-68fac8375746 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:28.951763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710d69b1-1cb5-42fe-b05d-68fac8375746 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:28.961164 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710d69b1-1cb5-42fe-b05d-68fac8375746 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:28.963497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710d69b1-1cb5-42fe-b05d-68fac8375746 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:28.980295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-710d69b1-1cb5-42fe-b05d-68fac8375746 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 30 09:27:28.980295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:28.980295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:28.980295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:28.980295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:28.980295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:28.980295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:28.980295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:28.980295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:28.980295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:28.980295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:28.980295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:28.980295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:28.980295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:28.980295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:28.980295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:28.981498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:28.981498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:28.981498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:28.981498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:28.981498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:28.981498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:28.981498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:28.981498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:28.981498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:28.981498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:28.981498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:28.981498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:28.981498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:28.981498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:28.981498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:28.981498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:28.981498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:28.981498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:28.983231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:28.983231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:28.983231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:28.983231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:28.983231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:28.983231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:28.983231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:28.983231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:28.983231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 30 09:27:28.983231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:28.986367 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 292/575] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:27:28 2022] GET /identity/v3/users/glance => generated 84 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:28.987393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710d69b1-1cb5-42fe-b05d-68fac8375746 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:29.038116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:29.063023 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9ef5b5-2bb8-415f-8f6b-8f062a2b931a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:29.097383 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710d69b1-1cb5-42fe-b05d-68fac8375746 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:29.100280 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a98c286-c1f3-49fa-80c9-df2b1e7e2938 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:29.100915 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a98c286-c1f3-49fa-80c9-df2b1e7e2938 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:29.101199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a98c286-c1f3-49fa-80c9-df2b1e7e2938 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:29.101476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a98c286-c1f3-49fa-80c9-df2b1e7e2938 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:29.109587 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a98c286-c1f3-49fa-80c9-df2b1e7e2938 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:29.111298 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a98c286-c1f3-49fa-80c9-df2b1e7e2938 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:29.123439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-6a98c286-c1f3-49fa-80c9-df2b1e7e2938 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 30 09:27:29.123439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:29.123439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:29.123439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:29.123439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:29.123439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:29.123439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:29.123439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:29.123439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:29.123439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:29.123439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:29.123439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:29.123439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:29.123439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:29.123439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:29.123439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:29.124554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:29.124554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:29.124554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:29.124554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:29.124554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:29.124554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:29.124554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:29.124554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:29.124554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:29.124554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:29.124554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:29.124554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:29.124554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:29.124554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:29.124554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:29.124554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:29.124554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:29.124554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:29.125537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:29.125537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:29.125537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:29.125537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:29.125537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:29.125537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:29.125537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:29.125537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:29.125537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 30 09:27:29.125537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:29.125537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 284/576] 10.222.0.39 () {56 vars in 1093 bytes} [Fri Sep 30 09:27:28 2022] GET /identity/v3/users/nova => generated 82 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:29.125537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a98c286-c1f3-49fa-80c9-df2b1e7e2938 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:29.136435 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912cce69-d472-4063-9823-7d06f9c32df8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:29.137040 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912cce69-d472-4063-9823-7d06f9c32df8 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:29.137316 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912cce69-d472-4063-9823-7d06f9c32df8 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:29.137583 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912cce69-d472-4063-9823-7d06f9c32df8 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:29.138786 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912cce69-d472-4063-9823-7d06f9c32df8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:29.139089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912cce69-d472-4063-9823-7d06f9c32df8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:29.140451 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912cce69-d472-4063-9823-7d06f9c32df8 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:29.152002 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 293/577] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:29.152798 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912cce69-d472-4063-9823-7d06f9c32df8 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:29.168662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:29.244743 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a98c286-c1f3-49fa-80c9-df2b1e7e2938 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:29.261774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912cce69-d472-4063-9823-7d06f9c32df8 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:29.286230 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa70d0f-5f52-4f52-a049-57e4d04f454c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:29.286871 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa70d0f-5f52-4f52-a049-57e4d04f454c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:29.287140 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa70d0f-5f52-4f52-a049-57e4d04f454c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:29.287406 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa70d0f-5f52-4f52-a049-57e4d04f454c None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:29.288121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa70d0f-5f52-4f52-a049-57e4d04f454c None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:29.288421 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa70d0f-5f52-4f52-a049-57e4d04f454c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:29.289852 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa70d0f-5f52-4f52-a049-57e4d04f454c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:29.292387 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:29.300274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5467a1bf-622e-4743-a58b-d78ebfa772f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:29.300899 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5467a1bf-622e-4743-a58b-d78ebfa772f6 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:29.301175 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5467a1bf-622e-4743-a58b-d78ebfa772f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:29.302032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5467a1bf-622e-4743-a58b-d78ebfa772f6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:29.307935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 285/578] 10.222.0.39 () {56 vars in 1137 bytes} [Fri Sep 30 09:27:29 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:29.308724 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa70d0f-5f52-4f52-a049-57e4d04f454c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:29.316833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5467a1bf-622e-4743-a58b-d78ebfa772f6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:29.321757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-5467a1bf-622e-4743-a58b-d78ebfa772f6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:29.321757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:29.321757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:29.321757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:29.321757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:29.321757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:29.321757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:29.321757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:29.321757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:29.321757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:29.321757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:29.321757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:29.321757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:29.321757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:29.321757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:29.321757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:29.326814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:29.326814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:29.326814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:29.326814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:29.326814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:29.326814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:29.326814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:29.326814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:29.326814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:29.326814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:29.326814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:29.326814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:29.326814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:29.326814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:29.326814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:29.326814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:29.327693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:29.327693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:29.327693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:29.327693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:29.327693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:29.327693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:29.327693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:29.327693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:29.327693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:29.327693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:29.327693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:29.327693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:29.327693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 294/579] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:29 2022] GET /identity/v3/roles/service => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:29.327693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5467a1bf-622e-4743-a58b-d78ebfa772f6 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:29.403064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:29.426636 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa70d0f-5f52-4f52-a049-57e4d04f454c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:29.434238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5467a1bf-622e-4743-a58b-d78ebfa772f6 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:29.462234 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6b008d-2f25-4677-84e4-88d8a02b11db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:29.462953 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b008d-2f25-4677-84e4-88d8a02b11db None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:29.463297 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b008d-2f25-4677-84e4-88d8a02b11db None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:29.463626 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b008d-2f25-4677-84e4-88d8a02b11db None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:29.464427 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6b008d-2f25-4677-84e4-88d8a02b11db None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:29.464803 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6b008d-2f25-4677-84e4-88d8a02b11db None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:29.466331 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6b008d-2f25-4677-84e4-88d8a02b11db None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:29.469456 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86a64dc-14fa-4383-b90c-b98a36f5ad98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:29.470135 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86a64dc-14fa-4383-b90c-b98a36f5ad98 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:29.470425 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86a64dc-14fa-4383-b90c-b98a36f5ad98 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:29.470729 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86a64dc-14fa-4383-b90c-b98a36f5ad98 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:29.478196 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86a64dc-14fa-4383-b90c-b98a36f5ad98 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:29.479719 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86a64dc-14fa-4383-b90c-b98a36f5ad98 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:29.485293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 286/580] 10.222.0.39 () {56 vars in 1133 bytes} [Fri Sep 30 09:27:29 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:29.486420 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6b008d-2f25-4677-84e4-88d8a02b11db None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:29.495659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-c86a64dc-14fa-4383-b90c-b98a36f5ad98 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 30 09:27:29.495659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:29.495659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:29.495659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:29.495659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:29.495659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:29.495659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:29.495659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:29.495659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:29.495659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:29.495659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:29.495659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:29.495659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:29.495659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:29.495659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:29.495659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:29.496686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:29.496686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:29.496686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:29.496686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:29.496686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:29.496686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:29.496686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:29.496686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:29.496686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:29.496686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:29.496686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:29.496686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:29.496686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:29.496686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:29.496686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:29.496686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:29.496686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:29.496686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:29.498215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:29.498215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:29.498215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:29.498215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:29.498215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:29.498215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:29.498215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:29.498215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:29.498215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 30 09:27:29.498215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:29.498215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 295/581] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:27:29 2022] GET /identity/v3/users/cinder => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:29.499544 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86a64dc-14fa-4383-b90c-b98a36f5ad98 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:29.547373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:29.596674 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6b008d-2f25-4677-84e4-88d8a02b11db None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:29.601373 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86a64dc-14fa-4383-b90c-b98a36f5ad98 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:29.631742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:29.633834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cd1c67-a79b-4740-955d-ccf532ab1781 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:29.634518 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cd1c67-a79b-4740-955d-ccf532ab1781 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:29.634826 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cd1c67-a79b-4740-955d-ccf532ab1781 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:29.635157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cd1c67-a79b-4740-955d-ccf532ab1781 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:29.636934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515b215f-efdd-4827-bb50-1d9786b79a1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:29.637531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515b215f-efdd-4827-bb50-1d9786b79a1a None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:29.638554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515b215f-efdd-4827-bb50-1d9786b79a1a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:29.638682 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515b215f-efdd-4827-bb50-1d9786b79a1a None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:29.639323 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515b215f-efdd-4827-bb50-1d9786b79a1a None admin] RBAC: Adding query filter params (name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:29.639624 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515b215f-efdd-4827-bb50-1d9786b79a1a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:29.641120 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515b215f-efdd-4827-bb50-1d9786b79a1a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:29.646518 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8cd1c67-a79b-4740-955d-ccf532ab1781 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:29.648074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8cd1c67-a79b-4740-955d-ccf532ab1781 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:29.650100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 296/582] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:27:29 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:29.651367 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515b215f-efdd-4827-bb50-1d9786b79a1a None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:29.664340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-a8cd1c67-a79b-4740-955d-ccf532ab1781 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:29.664340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:29.664340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:29.664340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:29.664340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:29.664340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:29.664340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:29.664340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:29.664340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:29.664340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:29.664340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:29.664340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:29.664340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:29.664340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:29.664340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:29.664340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:29.665311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:29.665311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:29.665311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:29.665311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:29.665311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:29.665311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:29.665311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:29.665311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:29.665311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:29.665311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:29.665311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:29.665311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:29.665311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:29.665311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:29.665311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:29.665311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:29.665311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:29.671947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:29.671947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:29.671947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:29.671947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 287/583] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:29.671947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cd1c67-a79b-4740-955d-ccf532ab1781 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:29.758788 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515b215f-efdd-4827-bb50-1d9786b79a1a None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:29.780266 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cd1c67-a79b-4740-955d-ccf532ab1781 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:29.783261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:29.793178 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75a4ce8-12d1-4cb9-b34b-2ad54984970e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:29.793807 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75a4ce8-12d1-4cb9-b34b-2ad54984970e None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:29.794200 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75a4ce8-12d1-4cb9-b34b-2ad54984970e None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:29.795003 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75a4ce8-12d1-4cb9-b34b-2ad54984970e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:29.804748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75a4ce8-12d1-4cb9-b34b-2ad54984970e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:29.809994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75a4ce8-12d1-4cb9-b34b-2ad54984970e None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:29.821326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-a75a4ce8-12d1-4cb9-b34b-2ad54984970e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:29.821326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:29.821326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:29.821326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:29.821326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:29.821326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:29.821326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:29.821326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:29.821326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:29.821326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:29.821326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:29.821326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:29.821326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:29.821326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:29.821326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:29.821326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:29.822289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:29.822289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:29.822289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:29.822289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:29.822289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:29.822289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:29.822289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:29.822289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:29.822289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:29.822289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:29.822289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:29.822289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:29.822289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:29.822289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:29.822289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:29.822289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:29.822289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:29.823406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:29.823406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:29.823406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:29.823406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 297/584] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:29.823406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75a4ce8-12d1-4cb9-b34b-2ad54984970e None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:29.828346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5059d55-9192-4a23-b5dc-65baf2dc0cc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:29.829169 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5059d55-9192-4a23-b5dc-65baf2dc0cc9 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:29.829620 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5059d55-9192-4a23-b5dc-65baf2dc0cc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:29.830072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5059d55-9192-4a23-b5dc-65baf2dc0cc9 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:29.831033 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5059d55-9192-4a23-b5dc-65baf2dc0cc9 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:29.831512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5059d55-9192-4a23-b5dc-65baf2dc0cc9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:29.833207 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5059d55-9192-4a23-b5dc-65baf2dc0cc9 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:29.853908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 288/585] 10.222.0.39 () {56 vars in 1137 bytes} [Fri Sep 30 09:27:29 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:29.854775 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5059d55-9192-4a23-b5dc-65baf2dc0cc9 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:29.877979 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:29.935818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75a4ce8-12d1-4cb9-b34b-2ad54984970e None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:29.963598 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5059d55-9192-4a23-b5dc-65baf2dc0cc9 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:29.971322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59000fc5-359c-495b-8199-bc41a4e44fef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:29.971948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59000fc5-359c-495b-8199-bc41a4e44fef None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:29.972227 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59000fc5-359c-495b-8199-bc41a4e44fef None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:29.972498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59000fc5-359c-495b-8199-bc41a4e44fef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:29.983224 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59000fc5-359c-495b-8199-bc41a4e44fef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:29.984666 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59000fc5-359c-495b-8199-bc41a4e44fef None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:29.998530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-59000fc5-359c-495b-8199-bc41a4e44fef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:29.998530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:29.998530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:29.998530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:29.998530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:29.998530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:29.998530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:29.998530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:29.998530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:29.998530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:29.998530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:29.998530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:29.998530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:29.998530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:29.998530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:29.998530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:29.999518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:29.999518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:29.999518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:29.999518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:29.999518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:29.999518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:29.999518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:29.999518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:29.999518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:29.999518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:29.999518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:29.999518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:29.999518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:29.999518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:29.999518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:29.999518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:29.999518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:30.000445 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:30.000445 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:30.000445 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:30.000445 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 298/586] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:30.000445 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59000fc5-359c-495b-8199-bc41a4e44fef None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:30.005755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e13bfd-3cf4-40e6-be60-14dcdc1a244e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:30.005755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e13bfd-3cf4-40e6-be60-14dcdc1a244e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:30.005987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e13bfd-3cf4-40e6-be60-14dcdc1a244e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:30.006173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e13bfd-3cf4-40e6-be60-14dcdc1a244e None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:30.007099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e13bfd-3cf4-40e6-be60-14dcdc1a244e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:30.007409 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e13bfd-3cf4-40e6-be60-14dcdc1a244e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:30.008751 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e13bfd-3cf4-40e6-be60-14dcdc1a244e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:30.022472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 289/587] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:30.023298 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e13bfd-3cf4-40e6-be60-14dcdc1a244e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:30.048638 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:30.103550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:30.107275 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59000fc5-359c-495b-8199-bc41a4e44fef None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:30.125284 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e13bfd-3cf4-40e6-be60-14dcdc1a244e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:30.142183 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e55e43-8411-4303-baff-53622f04ad6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:30.142917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e55e43-8411-4303-baff-53622f04ad6e None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:30.143291 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e55e43-8411-4303-baff-53622f04ad6e None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:30.143571 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e55e43-8411-4303-baff-53622f04ad6e None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:30.144253 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e55e43-8411-4303-baff-53622f04ad6e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:30.144557 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e55e43-8411-4303-baff-53622f04ad6e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:30.145919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e55e43-8411-4303-baff-53622f04ad6e None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:30.162879 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 299/588] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:30.162879 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e55e43-8411-4303-baff-53622f04ad6e None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:30.164808 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2214fe59-4165-4a36-8844-9b3f4f73ca4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:30.165521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2214fe59-4165-4a36-8844-9b3f4f73ca4d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:30.165807 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2214fe59-4165-4a36-8844-9b3f4f73ca4d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:30.166083 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2214fe59-4165-4a36-8844-9b3f4f73ca4d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:30.177475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2214fe59-4165-4a36-8844-9b3f4f73ca4d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:30.178964 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2214fe59-4165-4a36-8844-9b3f4f73ca4d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:30.196018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-2214fe59-4165-4a36-8844-9b3f4f73ca4d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:30.196018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:30.196018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:30.196018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:30.196018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:30.196018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:30.196018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:30.196018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:30.196018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:30.196018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:30.196018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:30.196018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:30.196018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:30.196018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:30.196018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:30.196018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:30.196956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:30.196956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:30.196956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:30.196956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:30.196956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:30.196956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:30.196956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:30.196956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:30.196956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:30.196956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:30.196956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:30.196956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:30.196956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:30.196956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:30.196956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:30.196956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:30.196956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:30.197941 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:30.197941 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:30.197941 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:30.197941 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 290/589] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:30.197941 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2214fe59-4165-4a36-8844-9b3f4f73ca4d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:30.278440 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:30.278672 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e55e43-8411-4303-baff-53622f04ad6e None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:30.304757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2214fe59-4165-4a36-8844-9b3f4f73ca4d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:30.317898 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d64be25-f361-484e-ba5d-d410acc86816 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:30.318502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d64be25-f361-484e-ba5d-d410acc86816 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:30.318805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d64be25-f361-484e-ba5d-d410acc86816 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:30.319076 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d64be25-f361-484e-ba5d-d410acc86816 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:30.319708 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d64be25-f361-484e-ba5d-d410acc86816 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:30.320005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d64be25-f361-484e-ba5d-d410acc86816 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:30.321373 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d64be25-f361-484e-ba5d-d410acc86816 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:30.338936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 300/590] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:30.338936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d64be25-f361-484e-ba5d-d410acc86816 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:30.346907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9c5325-1bf6-42db-97a6-6723beeef8f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:30.347653 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9c5325-1bf6-42db-97a6-6723beeef8f2 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:30.347851 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9c5325-1bf6-42db-97a6-6723beeef8f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:30.348101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9c5325-1bf6-42db-97a6-6723beeef8f2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:30.355035 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9c5325-1bf6-42db-97a6-6723beeef8f2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:30.356468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9c5325-1bf6-42db-97a6-6723beeef8f2 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:30.358568 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:30.368847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-4c9c5325-1bf6-42db-97a6-6723beeef8f2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:30.368847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:30.368847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:30.368847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:30.368847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:30.368847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:30.368847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:30.368847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:30.368847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:30.368847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:30.368847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:30.368847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:30.368847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:30.368847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:30.368847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:30.376108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:30.376108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:30.376108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:30.376108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:30.376108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:30.376108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:30.376108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:30.376108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:30.376108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:30.376108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:30.376108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:30.376108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:30.376108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:30.376108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:30.376108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:30.376108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:30.376108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:30.377210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:30.377210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:30.377210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:30.377210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:30.377210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:30.377210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:30.377210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:30.377210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:30.377210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 291/591] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:30.377210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9c5325-1bf6-42db-97a6-6723beeef8f2 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:30.452657 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d64be25-f361-484e-ba5d-d410acc86816 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:30.482179 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9c5325-1bf6-42db-97a6-6723beeef8f2 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:30.486431 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decf77be-63e6-4fcd-be70-79fd63e9d514 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:30.487085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decf77be-63e6-4fcd-be70-79fd63e9d514 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:30.487361 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decf77be-63e6-4fcd-be70-79fd63e9d514 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:30.487629 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decf77be-63e6-4fcd-be70-79fd63e9d514 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:30.494256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decf77be-63e6-4fcd-be70-79fd63e9d514 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:30.495741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decf77be-63e6-4fcd-be70-79fd63e9d514 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:30.507595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-decf77be-63e6-4fcd-be70-79fd63e9d514 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:30.507595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:30.507595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:30.507595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:30.507595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:30.507595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:30.507595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:30.507595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:30.507595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:30.507595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:30.507595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:30.507595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:30.507595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:30.507595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:30.507595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:30.508506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:30.508506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:30.508506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:30.508506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:30.508506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:30.508506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:30.508506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:30.508506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:30.508506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:30.508506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:30.508506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:30.508506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:30.508506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:30.508506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:30.508506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:30.508506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:30.508506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:30.509457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:30.509457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:30.509457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:30.509457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:30.509457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:30.509457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:30.509457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:30.509457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:30.509457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 301/592] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:30.509457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decf77be-63e6-4fcd-be70-79fd63e9d514 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:30.522260 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0586c8fc-d11f-4ab8-a617-9cae475086b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:30.522899 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0586c8fc-d11f-4ab8-a617-9cae475086b7 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:30.523173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0586c8fc-d11f-4ab8-a617-9cae475086b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:30.523444 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0586c8fc-d11f-4ab8-a617-9cae475086b7 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:30.524075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0586c8fc-d11f-4ab8-a617-9cae475086b7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:30.524374 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0586c8fc-d11f-4ab8-a617-9cae475086b7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:30.525729 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0586c8fc-d11f-4ab8-a617-9cae475086b7 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:30.527245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:30.538093 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 292/593] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:30.538972 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0586c8fc-d11f-4ab8-a617-9cae475086b7 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:30.594268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:30.618938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decf77be-63e6-4fcd-be70-79fd63e9d514 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:30.643124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0586c8fc-d11f-4ab8-a617-9cae475086b7 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:30.652157 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb8e5f4-3d8a-4ee6-8d5c-fa71d7f629b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:30.652807 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb8e5f4-3d8a-4ee6-8d5c-fa71d7f629b3 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:30.653082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb8e5f4-3d8a-4ee6-8d5c-fa71d7f629b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:30.653353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb8e5f4-3d8a-4ee6-8d5c-fa71d7f629b3 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:30.661835 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb8e5f4-3d8a-4ee6-8d5c-fa71d7f629b3 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:30.663369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb8e5f4-3d8a-4ee6-8d5c-fa71d7f629b3 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:30.676506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-3fb8e5f4-3d8a-4ee6-8d5c-fa71d7f629b3 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 30 09:27:30.676506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:30.676506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:30.676506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:30.676506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:30.676506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:30.676506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:30.676506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:30.676506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:30.676506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:30.676506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:30.676506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:30.676506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:30.676506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:30.676506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:30.676506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:30.677559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:30.677559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:30.677559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:30.677559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:30.677559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:30.677559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:30.677559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:30.677559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:30.677559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:30.677559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:30.677559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:30.677559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:30.677559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:30.677559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:30.677559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:30.677559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:30.677559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:30.677559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:30.678554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:30.678554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:30.678554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:30.678554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:30.678554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:30.678554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:30.678554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:30.678554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:30.678554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 30 09:27:30.678554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:30.678554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 302/594] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:30 2022] GET /identity/v3/users/neutron => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:30.678554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb8e5f4-3d8a-4ee6-8d5c-fa71d7f629b3 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:30.689333 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bd6666-d16a-46a9-b184-1ecbcafe9bfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:30.690183 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bd6666-d16a-46a9-b184-1ecbcafe9bfa None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:30.690463 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bd6666-d16a-46a9-b184-1ecbcafe9bfa None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:30.690780 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bd6666-d16a-46a9-b184-1ecbcafe9bfa None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:30.691453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bd6666-d16a-46a9-b184-1ecbcafe9bfa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:30.691893 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bd6666-d16a-46a9-b184-1ecbcafe9bfa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:30.693557 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bd6666-d16a-46a9-b184-1ecbcafe9bfa None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:30.711683 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 293/595] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:30.711683 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bd6666-d16a-46a9-b184-1ecbcafe9bfa None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:30.773808 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:30.785431 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb8e5f4-3d8a-4ee6-8d5c-fa71d7f629b3 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:30.818479 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bd6666-d16a-46a9-b184-1ecbcafe9bfa None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:30.819199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f44eea-919d-45eb-ad48-f4d890ec3ef5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:30.819801 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f44eea-919d-45eb-ad48-f4d890ec3ef5 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:30.820088 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f44eea-919d-45eb-ad48-f4d890ec3ef5 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:30.820359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f44eea-919d-45eb-ad48-f4d890ec3ef5 None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:30.821039 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f44eea-919d-45eb-ad48-f4d890ec3ef5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:30.821345 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f44eea-919d-45eb-ad48-f4d890ec3ef5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:30.822807 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f44eea-919d-45eb-ad48-f4d890ec3ef5 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:30.829953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:30.834809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 303/596] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:30.835667 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f44eea-919d-45eb-ad48-f4d890ec3ef5 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:30.852398 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373a2927-874d-4beb-8f8a-3d410031588c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:30.853023 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373a2927-874d-4beb-8f8a-3d410031588c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:30.853306 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373a2927-874d-4beb-8f8a-3d410031588c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:30.853579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373a2927-874d-4beb-8f8a-3d410031588c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:30.860783 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373a2927-874d-4beb-8f8a-3d410031588c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:30.862226 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373a2927-874d-4beb-8f8a-3d410031588c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:30.873922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-373a2927-874d-4beb-8f8a-3d410031588c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:30.873922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:30.873922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:30.873922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:30.873922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:30.873922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:30.873922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:30.873922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:30.873922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:30.873922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:30.873922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:30.873922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:30.873922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:30.873922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:30.873922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:30.875008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:30.875008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:30.875008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:30.875008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:30.875008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:30.875008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:30.875008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:30.875008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:30.875008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:30.875008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:30.875008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:30.875008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:30.875008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:30.875008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:30.875008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:30.875008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:30.875008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:30.876332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:30.876332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:30.876332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:30.876332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:30.876332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:30.876332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:30.876332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:30.876332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:30.876332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 294/597] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:30.876332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373a2927-874d-4beb-8f8a-3d410031588c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:30.934203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f44eea-919d-45eb-ad48-f4d890ec3ef5 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:30.967779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23654914-282d-468d-ad35-fbdeb2a616b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:30.968309 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23654914-282d-468d-ad35-fbdeb2a616b9 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:30.968585 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23654914-282d-468d-ad35-fbdeb2a616b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:30.968852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23654914-282d-468d-ad35-fbdeb2a616b9 None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:30.969562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23654914-282d-468d-ad35-fbdeb2a616b9 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:30.969861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23654914-282d-468d-ad35-fbdeb2a616b9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:30.971590 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23654914-282d-468d-ad35-fbdeb2a616b9 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:30.972742 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373a2927-874d-4beb-8f8a-3d410031588c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:30.987032 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:30.990134 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 304/598] 10.222.0.39 () {56 vars in 1139 bytes} [Fri Sep 30 09:27:30 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:30.991392 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90725082-1078-4cc0-850d-2705b962c015 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:30.991748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90725082-1078-4cc0-850d-2705b962c015 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:30.992115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90725082-1078-4cc0-850d-2705b962c015 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:30.993478 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 305/599] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:30.994068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90725082-1078-4cc0-850d-2705b962c015 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:31.009582 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b22d84-a2e1-457d-b41e-224230e80005 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:31.010242 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b22d84-a2e1-457d-b41e-224230e80005 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:31.010525 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b22d84-a2e1-457d-b41e-224230e80005 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:31.010836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b22d84-a2e1-457d-b41e-224230e80005 None admin] PATH_INFO: `/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/bf907d4033064fb8a2707b1bbb0a180b/roles/144dd855f74945548ea65db37acda1c1` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:31.025215 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b22d84-a2e1-457d-b41e-224230e80005 None admin] RBAC: Authorizing `identity:create_grant(project_id=c3802ced1fce4997a016d41eca843ef8, user_id=bf907d4033064fb8a2707b1bbb0a180b, role_id=144dd855f74945548ea65db37acda1c1)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:31.026673 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b22d84-a2e1-457d-b41e-224230e80005 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:31.049651 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:31.091642 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90725082-1078-4cc0-850d-2705b962c015 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:31.093010 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 295/600] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:27:30 2022] PUT /identity/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/bf907d4033064fb8a2707b1bbb0a180b/roles/144dd855f74945548ea65db37acda1c1 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:27:31.093818 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b22d84-a2e1-457d-b41e-224230e80005 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:31.130123 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c65d6b-ce28-4e91-9e78-ff794d80e085 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:31.130829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c65d6b-ce28-4e91-9e78-ff794d80e085 None admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:31.131103 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c65d6b-ce28-4e91-9e78-ff794d80e085 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:31.131376 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c65d6b-ce28-4e91-9e78-ff794d80e085 None admin] PATH_INFO: `/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/7ff68fb87700423ab53c90520a6bd3b9/roles/144dd855f74945548ea65db37acda1c1` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:31.148565 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c65d6b-ce28-4e91-9e78-ff794d80e085 None admin] RBAC: Authorizing `identity:create_grant(project_id=c3802ced1fce4997a016d41eca843ef8, user_id=7ff68fb87700423ab53c90520a6bd3b9, role_id=144dd855f74945548ea65db37acda1c1)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:31.150432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c65d6b-ce28-4e91-9e78-ff794d80e085 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:31.207748 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b22d84-a2e1-457d-b41e-224230e80005 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:31.234154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 306/601] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:27:30 2022] PUT /identity/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/7ff68fb87700423ab53c90520a6bd3b9/roles/144dd855f74945548ea65db37acda1c1 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:27:31.244480 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c65d6b-ce28-4e91-9e78-ff794d80e085 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:31.252953 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ec0c18-796e-4f17-a2d7-3967986494cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:31.254257 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ec0c18-796e-4f17-a2d7-3967986494cf None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:31.254865 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ec0c18-796e-4f17-a2d7-3967986494cf None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:31.256117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ec0c18-796e-4f17-a2d7-3967986494cf None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:31.257028 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ec0c18-796e-4f17-a2d7-3967986494cf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:31.257793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ec0c18-796e-4f17-a2d7-3967986494cf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:31.260402 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ec0c18-796e-4f17-a2d7-3967986494cf None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:31.279791 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 296/602] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:31.281249 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb73fd8-2e13-4b18-af90-75e25cfcf161 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:31.281591 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb73fd8-2e13-4b18-af90-75e25cfcf161 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:31.281907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb73fd8-2e13-4b18-af90-75e25cfcf161 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:31.310395 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-0cb73fd8-2e13-4b18-af90-75e25cfcf161 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:31.312356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0cb73fd8-2e13-4b18-af90-75e25cfcf161 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:31.350514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:31.381279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c65d6b-ce28-4e91-9e78-ff794d80e085 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:31.392504 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:31.420590 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e0b1a4-5fe0-4551-bae1-fcd290ff9ce0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:31.421330 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e0b1a4-5fe0-4551-bae1-fcd290ff9ce0 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:31.421680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e0b1a4-5fe0-4551-bae1-fcd290ff9ce0 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:31.422020 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e0b1a4-5fe0-4551-bae1-fcd290ff9ce0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:31.423018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 297/603] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:31.424034 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb73fd8-2e13-4b18-af90-75e25cfcf161 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:31.465538 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e0b1a4-5fe0-4551-bae1-fcd290ff9ce0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:31.468711 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e0b1a4-5fe0-4551-bae1-fcd290ff9ce0 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:31.508533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-08e0b1a4-5fe0-4551-bae1-fcd290ff9ce0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:31.508533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:31.508533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:31.508533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:31.508533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:31.508533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:31.508533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:31.508533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:31.508533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:31.508533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:31.508533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:31.508533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:31.508533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:31.508533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:31.508533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:31.508533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:31.509758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:31.509758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:31.509758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:31.509758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:31.509758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:31.509758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:31.509758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:31.509758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:31.509758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:31.509758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:31.509758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:31.509758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:31.509758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:31.509758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:31.509758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:31.509758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:31.509758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:31.510873 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:31.510873 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:31.510873 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:31.510873 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 307/604] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:31.520036 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e0b1a4-5fe0-4551-bae1-fcd290ff9ce0 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:31.543808 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb73fd8-2e13-4b18-af90-75e25cfcf161 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:31.582530 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5032c7f3-543b-432b-a1db-c13dfcd463cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:31.583481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5032c7f3-543b-432b-a1db-c13dfcd463cc None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:31.583913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5032c7f3-543b-432b-a1db-c13dfcd463cc None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:31.584342 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5032c7f3-543b-432b-a1db-c13dfcd463cc None admin] PATH_INFO: `/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/4e2742f567444dff88ed919be52217de/roles/144dd855f74945548ea65db37acda1c1` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:31.600275 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5032c7f3-543b-432b-a1db-c13dfcd463cc None admin] RBAC: Authorizing `identity:create_grant(project_id=c3802ced1fce4997a016d41eca843ef8, user_id=4e2742f567444dff88ed919be52217de, role_id=144dd855f74945548ea65db37acda1c1)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:31.601917 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5032c7f3-543b-432b-a1db-c13dfcd463cc None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:31.703399 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e0b1a4-5fe0-4551-bae1-fcd290ff9ce0 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:31.725585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 298/605] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:27:31 2022] PUT /identity/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/4e2742f567444dff88ed919be52217de/roles/144dd855f74945548ea65db37acda1c1 => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:27:31.763616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27521600-d38b-4c05-a7b0-ed823d9652e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:31.764519 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27521600-d38b-4c05-a7b0-ed823d9652e3 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:31.764993 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27521600-d38b-4c05-a7b0-ed823d9652e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:31.765458 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27521600-d38b-4c05-a7b0-ed823d9652e3 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:31.766688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27521600-d38b-4c05-a7b0-ed823d9652e3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:31.767284 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27521600-d38b-4c05-a7b0-ed823d9652e3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:31.769987 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27521600-d38b-4c05-a7b0-ed823d9652e3 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:31.792503 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 308/606] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:31.800303 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5032c7f3-543b-432b-a1db-c13dfcd463cc None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:31.815987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:31.980523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5032c7f3-543b-432b-a1db-c13dfcd463cc None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:32.022560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27521600-d38b-4c05-a7b0-ed823d9652e3 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:32.040790 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1853b105-2282-4ba3-bb89-cb351db7e0b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:32.041794 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1853b105-2282-4ba3-bb89-cb351db7e0b6 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:32.042288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1853b105-2282-4ba3-bb89-cb351db7e0b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:32.043366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1853b105-2282-4ba3-bb89-cb351db7e0b6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:32.054991 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1853b105-2282-4ba3-bb89-cb351db7e0b6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:32.058579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1853b105-2282-4ba3-bb89-cb351db7e0b6 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:32.062387 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:32.086877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-1853b105-2282-4ba3-bb89-cb351db7e0b6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:32.086877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:32.086877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:32.086877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:32.086877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:32.086877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:32.086877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:32.086877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:32.086877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:32.086877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:32.086877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:32.086877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:32.086877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:32.086877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:32.086877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:32.088764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:32.088764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:32.088764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:32.088764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:32.088764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:32.088764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:32.088764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:32.088764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:32.088764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:32.088764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:32.088764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:32.088764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:32.088764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:32.088764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:32.088764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:32.088764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:32.088764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:32.090399 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:32.090399 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:32.090399 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:32.090399 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:32.090399 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:32.090399 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:32.090399 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:32.090399 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:32.090399 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 299/607] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:32.099630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1853b105-2282-4ba3-bb89-cb351db7e0b6 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:32.167873 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:32.217390 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27521600-d38b-4c05-a7b0-ed823d9652e3 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:32.234838 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1853b105-2282-4ba3-bb89-cb351db7e0b6 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:32.296113 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c16206-569b-4754-a422-71e538ed65da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:32.296715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c16206-569b-4754-a422-71e538ed65da None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:32.296999 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c16206-569b-4754-a422-71e538ed65da None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:32.297265 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c16206-569b-4754-a422-71e538ed65da None admin] PATH_INFO: `/v3/roles/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:32.306528 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa57807-9591-4e7f-a61b-d9e6b578ce12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:32.307197 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa57807-9591-4e7f-a61b-d9e6b578ce12 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:32.307472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa57807-9591-4e7f-a61b-d9e6b578ce12 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:32.307745 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa57807-9591-4e7f-a61b-d9e6b578ce12 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:32.308508 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa57807-9591-4e7f-a61b-d9e6b578ce12 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:32.308817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa57807-9591-4e7f-a61b-d9e6b578ce12 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:32.310871 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa57807-9591-4e7f-a61b-d9e6b578ce12 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:32.324992 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 300/608] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:32.327604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c16206-569b-4754-a422-71e538ed65da None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:32.337176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-68c16206-569b-4754-a422-71e538ed65da None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:32.337176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:32.337176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:32.337176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:32.337176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:32.337176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:32.337176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:32.337176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:32.337176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:32.337176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:32.337176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:32.337176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:32.337176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:32.337176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:32.337176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:32.337176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:32.338236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:32.338236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:32.338236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:32.338236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:32.338236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:32.338236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:32.338236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:32.338236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:32.338236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:32.338236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:32.338236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:32.338236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:32.338236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:32.338236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:32.338236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:32.338236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:32.339137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:32.339137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:32.339137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:32.339137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:32.339137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:32.339137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:32.339137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:32.339137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:32.339137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:32.339137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:32.339137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:32.339137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:32.345430 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 309/609] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 324 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:32.345847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa57807-9591-4e7f-a61b-d9e6b578ce12 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:32.354259 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c16206-569b-4754-a422-71e538ed65da None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:32.476438 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa57807-9591-4e7f-a61b-d9e6b578ce12 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:32.523964 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:32.537465 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc3d029-c2af-4052-ad37-097eba6ae251 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:32.538336 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc3d029-c2af-4052-ad37-097eba6ae251 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:32.538792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc3d029-c2af-4052-ad37-097eba6ae251 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:32.539287 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc3d029-c2af-4052-ad37-097eba6ae251 None admin] PATH_INFO: `/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/9266babd21414ae9998716e5a0478e9d/roles/144dd855f74945548ea65db37acda1c1` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:32.542430 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:32.567800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc3d029-c2af-4052-ad37-097eba6ae251 None admin] RBAC: Authorizing `identity:create_grant(project_id=c3802ced1fce4997a016d41eca843ef8, user_id=9266babd21414ae9998716e5a0478e9d, role_id=144dd855f74945548ea65db37acda1c1)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:32.567800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc3d029-c2af-4052-ad37-097eba6ae251 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:32.579035 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c16206-569b-4754-a422-71e538ed65da None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:32.616315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5832bab-1507-43c3-adb6-903ab675ab6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:32.617146 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5832bab-1507-43c3-adb6-903ab675ab6a None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:32.617616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5832bab-1507-43c3-adb6-903ab675ab6a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:32.618093 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5832bab-1507-43c3-adb6-903ab675ab6a None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:32.618939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5832bab-1507-43c3-adb6-903ab675ab6a None admin] RBAC: Adding query filter params (name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:32.619385 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5832bab-1507-43c3-adb6-903ab675ab6a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:32.621017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5832bab-1507-43c3-adb6-903ab675ab6a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:32.629743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 310/610] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:27:32 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:32.642052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5832bab-1507-43c3-adb6-903ab675ab6a None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:32.644461 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 301/611] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:27:32 2022] PUT /identity/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/9266babd21414ae9998716e5a0478e9d/roles/144dd855f74945548ea65db37acda1c1 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:27:32.758032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5832bab-1507-43c3-adb6-903ab675ab6a None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:32.795776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564915d3-3c3c-4b76-9fac-2ea8e7bcadab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:32.797165 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564915d3-3c3c-4b76-9fac-2ea8e7bcadab None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:32.797594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564915d3-3c3c-4b76-9fac-2ea8e7bcadab None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:32.798014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564915d3-3c3c-4b76-9fac-2ea8e7bcadab None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:32.808070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:32.810243 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564915d3-3c3c-4b76-9fac-2ea8e7bcadab None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:32.811837 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564915d3-3c3c-4b76-9fac-2ea8e7bcadab None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:32.827883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-564915d3-3c3c-4b76-9fac-2ea8e7bcadab None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:32.827883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:32.827883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:32.827883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:32.827883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:32.827883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:32.827883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:32.827883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:32.827883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:32.827883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:32.827883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:32.827883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:32.827883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:32.827883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:32.827883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:32.827883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:32.829874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:32.829874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:32.829874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:32.829874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:32.829874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:32.829874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:32.829874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:32.829874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:32.829874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:32.829874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:32.829874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:32.829874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:32.829874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:32.829874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:32.829874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:32.829874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:32.829874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:32.831748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:32.831748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:32.831748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:32.831748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 311/612] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:32.840604 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc3d029-c2af-4052-ad37-097eba6ae251 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:32.970519 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc3d029-c2af-4052-ad37-097eba6ae251 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:33.008431 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e25515-4b0c-4767-877c-0ebf68a4fca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:33.009376 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e25515-4b0c-4767-877c-0ebf68a4fca4 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:33.009376 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e25515-4b0c-4767-877c-0ebf68a4fca4 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:33.009652 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e25515-4b0c-4767-877c-0ebf68a4fca4 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:33.010169 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e25515-4b0c-4767-877c-0ebf68a4fca4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:33.010472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e25515-4b0c-4767-877c-0ebf68a4fca4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:33.011901 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e25515-4b0c-4767-877c-0ebf68a4fca4 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:33.027525 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 302/613] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:33.036176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564915d3-3c3c-4b76-9fac-2ea8e7bcadab None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:33.170421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564915d3-3c3c-4b76-9fac-2ea8e7bcadab None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:33.211952 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5bd2e1-a5b9-49cd-9e39-1a71e3d66bda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:33.212590 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5bd2e1-a5b9-49cd-9e39-1a71e3d66bda None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:33.212878 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5bd2e1-a5b9-49cd-9e39-1a71e3d66bda None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:33.213120 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5bd2e1-a5b9-49cd-9e39-1a71e3d66bda None admin] PATH_INFO: `/v3/users/swift` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:33.224518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5bd2e1-a5b9-49cd-9e39-1a71e3d66bda None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:33.225964 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5bd2e1-a5b9-49cd-9e39-1a71e3d66bda None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:33.241322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-fd5bd2e1-a5b9-49cd-9e39-1a71e3d66bda None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 30 09:27:33.241322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:33.241322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:33.241322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:33.241322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:33.241322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:33.241322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:33.241322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:33.241322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:33.241322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:33.241322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:33.241322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:33.241322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:33.241322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:33.241322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:33.241322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:33.243085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:33.243085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:33.243085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:33.243085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:33.243085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:33.243085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:33.243085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:33.243085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:33.243085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:33.243085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:33.243085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:33.243085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:33.243085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:33.243085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:33.243085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:33.243085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:33.243085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:33.243085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:33.244937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:33.244937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:33.244937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:33.244937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:33.244937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:33.244937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:33.244937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:33.244937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:33.244937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 30 09:27:33.244937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:33.244937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 312/614] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:27:33 2022] GET /identity/v3/users/swift => generated 83 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:33.251350 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e25515-4b0c-4767-877c-0ebf68a4fca4 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:33.326986 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:33.414703 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e25515-4b0c-4767-877c-0ebf68a4fca4 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:33.458162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d832688-2c37-41bd-9f0a-02e16987acd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:33.459057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d832688-2c37-41bd-9f0a-02e16987acd3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:33.459515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d832688-2c37-41bd-9f0a-02e16987acd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:33.459969 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d832688-2c37-41bd-9f0a-02e16987acd3 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:33.460855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d832688-2c37-41bd-9f0a-02e16987acd3 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:33.461358 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d832688-2c37-41bd-9f0a-02e16987acd3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:33.462984 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d832688-2c37-41bd-9f0a-02e16987acd3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:33.489670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 303/615] 10.222.0.39 () {56 vars in 1135 bytes} [Fri Sep 30 09:27:33 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:33.499049 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5bd2e1-a5b9-49cd-9e39-1a71e3d66bda None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:33.626574 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:33.706281 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5bd2e1-a5b9-49cd-9e39-1a71e3d66bda None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:33.756251 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb6e7a2-fd90-4b5f-b020-a8febb25b551 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:33.757874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb6e7a2-fd90-4b5f-b020-a8febb25b551 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:33.758156 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb6e7a2-fd90-4b5f-b020-a8febb25b551 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:33.758431 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb6e7a2-fd90-4b5f-b020-a8febb25b551 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:33.774158 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb6e7a2-fd90-4b5f-b020-a8febb25b551 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:33.775643 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb6e7a2-fd90-4b5f-b020-a8febb25b551 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:33.794920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-2fb6e7a2-fd90-4b5f-b020-a8febb25b551 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:33.794920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:33.794920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:33.794920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:33.794920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:33.794920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:33.794920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:33.794920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:33.794920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:33.794920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:33.794920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:33.794920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:33.794920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:33.794920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:33.794920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:33.794920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:33.797694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:33.797694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:33.797694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:33.797694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:33.797694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:33.797694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:33.797694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:33.797694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:33.797694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:33.797694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:33.797694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:33.797694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:33.797694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:33.797694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:33.797694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:33.797694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:33.797694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:33.799769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:33.799769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:33.799769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:33.799769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 313/616] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:33.808492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d832688-2c37-41bd-9f0a-02e16987acd3 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:33.962521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:33.978820 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d832688-2c37-41bd-9f0a-02e16987acd3 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:34.035091 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae50753-6028-4c41-b779-6c20ec2f02c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:34.035689 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae50753-6028-4c41-b779-6c20ec2f02c1 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:34.035932 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae50753-6028-4c41-b779-6c20ec2f02c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:34.036151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae50753-6028-4c41-b779-6c20ec2f02c1 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:34.036794 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae50753-6028-4c41-b779-6c20ec2f02c1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:34.037096 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae50753-6028-4c41-b779-6c20ec2f02c1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:34.038471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae50753-6028-4c41-b779-6c20ec2f02c1 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:34.051176 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 304/617] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:34.056695 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb6e7a2-fd90-4b5f-b020-a8febb25b551 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:34.191108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb6e7a2-fd90-4b5f-b020-a8febb25b551 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:34.193512 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:34.239298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0140fcbf-d956-4e20-8c33-a355be8d9518 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:34.240361 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0140fcbf-d956-4e20-8c33-a355be8d9518 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:34.240823 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0140fcbf-d956-4e20-8c33-a355be8d9518 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:34.241273 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0140fcbf-d956-4e20-8c33-a355be8d9518 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:34.248863 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0140fcbf-d956-4e20-8c33-a355be8d9518 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:34.250523 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0140fcbf-d956-4e20-8c33-a355be8d9518 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:34.265976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-0140fcbf-d956-4e20-8c33-a355be8d9518 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:34.265976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:34.265976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:34.265976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:34.265976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:34.265976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:34.265976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:34.265976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:34.265976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:34.265976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:34.265976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:34.265976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:34.265976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:34.265976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:34.265976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:34.266958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:34.266958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:34.266958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:34.266958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:34.266958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:34.266958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:34.266958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:34.266958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:34.266958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:34.266958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:34.266958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:34.266958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:34.266958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:34.266958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:34.266958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:34.266958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:34.266958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:34.267870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:34.267870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:34.267870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:34.267870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:34.267870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:34.267870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:34.267870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:34.267870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:34.269861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 314/618] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:34.279707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae50753-6028-4c41-b779-6c20ec2f02c1 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:34.391261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae50753-6028-4c41-b779-6c20ec2f02c1 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:34.426866 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6832733c-4c35-430d-8154-1fecaeeaca40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:34.427496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6832733c-4c35-430d-8154-1fecaeeaca40 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:34.427777 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6832733c-4c35-430d-8154-1fecaeeaca40 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:34.428049 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6832733c-4c35-430d-8154-1fecaeeaca40 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:34.428750 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6832733c-4c35-430d-8154-1fecaeeaca40 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:34.429059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6832733c-4c35-430d-8154-1fecaeeaca40 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:34.430539 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6832733c-4c35-430d-8154-1fecaeeaca40 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:34.444697 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 305/619] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:34.451235 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0140fcbf-d956-4e20-8c33-a355be8d9518 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:34.560908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddb98eb-ad1a-4ae1-a6f2-f1b6eb0dcd76 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:34.561248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddb98eb-ad1a-4ae1-a6f2-f1b6eb0dcd76 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:34.561527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddb98eb-ad1a-4ae1-a6f2-f1b6eb0dcd76 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:34.563169 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 306/620] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:34.568603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626fcb3a-6e1d-4314-ae7b-31ed66dcb9a2 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:34.569118 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626fcb3a-6e1d-4314-ae7b-31ed66dcb9a2 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:34.569615 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626fcb3a-6e1d-4314-ae7b-31ed66dcb9a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:34.588975 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-626fcb3a-6e1d-4314-ae7b-31ed66dcb9a2 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:34.590687 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-626fcb3a-6e1d-4314-ae7b-31ed66dcb9a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:34.602894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:34.614902 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0140fcbf-d956-4e20-8c33-a355be8d9518 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:34.687924 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a507a5d-16a9-4696-ba58-35649250aa4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:34.688497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a507a5d-16a9-4696-ba58-35649250aa4a None admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:34.688770 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a507a5d-16a9-4696-ba58-35649250aa4a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:34.689036 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a507a5d-16a9-4696-ba58-35649250aa4a None admin] PATH_INFO: `/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/0e5ad921e6fa41c297696ec7279f0fc4/roles/144dd855f74945548ea65db37acda1c1` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:34.707908 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a507a5d-16a9-4696-ba58-35649250aa4a None admin] RBAC: Authorizing `identity:create_grant(project_id=c3802ced1fce4997a016d41eca843ef8, user_id=0e5ad921e6fa41c297696ec7279f0fc4, role_id=144dd855f74945548ea65db37acda1c1)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:34.709361 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a507a5d-16a9-4696-ba58-35649250aa4a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:34.723501 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 307/621] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:27:34.761947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae1b7a4-0ff0-4daf-9e91-5e81d259cd7a None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:34.762284 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae1b7a4-0ff0-4daf-9e91-5e81d259cd7a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:34.763140 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae1b7a4-0ff0-4daf-9e91-5e81d259cd7a None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:34.764838 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 308/622] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:34.770114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a36992-8735-44b6-b130-58117e236b5c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:34.770399 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a36992-8735-44b6-b130-58117e236b5c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:34.770719 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a36992-8735-44b6-b130-58117e236b5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:34.793685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-42a36992-8735-44b6-b130-58117e236b5c None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:34.798898 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-42a36992-8735-44b6-b130-58117e236b5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:34.799003 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 315/623] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:27:34 2022] PUT /identity/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/0e5ad921e6fa41c297696ec7279f0fc4/roles/144dd855f74945548ea65db37acda1c1 => generated 0 bytes in 347 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:27:34.898883 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 309/624] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:27:34.981589 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29876fb3-a34c-4add-a23b-59914fe52761 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:34.981931 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29876fb3-a34c-4add-a23b-59914fe52761 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:34.982211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29876fb3-a34c-4add-a23b-59914fe52761 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:34.983885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 316/625] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:34.989298 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68044ac6-639e-438d-a516-444ed8ce8c14 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:34.989586 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68044ac6-639e-438d-a516-444ed8ce8c14 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:34.989867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68044ac6-639e-438d-a516-444ed8ce8c14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:35.009806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-68044ac6-639e-438d-a516-444ed8ce8c14 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:35.011307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-68044ac6-639e-438d-a516-444ed8ce8c14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:35.055073 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29876fb3-a34c-4add-a23b-59914fe52761 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:35.085133 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:35.111642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 310/626] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:35.180289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29876fb3-a34c-4add-a23b-59914fe52761 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:35.217540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0930c9-b9c3-4b8d-8661-c71b785665b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:35.218176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0930c9-b9c3-4b8d-8661-c71b785665b5 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:35.218456 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0930c9-b9c3-4b8d-8661-c71b785665b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:35.218755 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0930c9-b9c3-4b8d-8661-c71b785665b5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:35.233975 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0930c9-b9c3-4b8d-8661-c71b785665b5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:35.238917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-ce0930c9-b9c3-4b8d-8661-c71b785665b5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:35.238917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:35.238917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:35.238917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:35.238917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:35.238917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:35.238917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:35.238917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:35.238917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:35.238917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:35.238917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:35.238917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:35.238917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:35.238917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:35.238917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:35.238917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:35.240054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:35.240054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:35.240054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:35.240054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:35.240054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:35.240054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:35.240054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:35.240054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:35.240054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:35.240054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:35.240054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:35.240054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:35.240054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:35.240054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:35.240054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:35.240054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:35.241105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:35.241105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:35.241105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:35.241105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:35.241105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:35.241105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:35.241105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:35.241105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:35.241105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:35.241105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:35.241105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:35.241105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:35.241105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 317/627] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:35.246256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68044ac6-639e-438d-a516-444ed8ce8c14 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:35.250894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:35.272739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0930c9-b9c3-4b8d-8661-c71b785665b5 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:35.413563 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:35.423463 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68044ac6-639e-438d-a516-444ed8ce8c14 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:35.441808 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0930c9-b9c3-4b8d-8661-c71b785665b5 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:35.463122 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23d77ca-b3b3-45e5-b0d0-5b778a1dcce8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:35.463828 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23d77ca-b3b3-45e5-b0d0-5b778a1dcce8 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:35.464067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23d77ca-b3b3-45e5-b0d0-5b778a1dcce8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:35.464343 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23d77ca-b3b3-45e5-b0d0-5b778a1dcce8 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:35.464999 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23d77ca-b3b3-45e5-b0d0-5b778a1dcce8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:35.465301 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23d77ca-b3b3-45e5-b0d0-5b778a1dcce8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:35.467229 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23d77ca-b3b3-45e5-b0d0-5b778a1dcce8 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:35.476006 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44fdb0b-a0fb-4890-bef0-4bf1f92a70ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:35.476628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44fdb0b-a0fb-4890-bef0-4bf1f92a70ed None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:35.476903 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44fdb0b-a0fb-4890-bef0-4bf1f92a70ed None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:35.477173 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44fdb0b-a0fb-4890-bef0-4bf1f92a70ed None admin] PATH_INFO: `/v3/roles/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:35.478741 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 311/628] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:27:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:35.480051 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056b4a8e-aa3e-4421-aeac-c24c0fba3942 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:35.480403 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056b4a8e-aa3e-4421-aeac-c24c0fba3942 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:35.480936 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056b4a8e-aa3e-4421-aeac-c24c0fba3942 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:35.482664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 312/629] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:35.483471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056b4a8e-aa3e-4421-aeac-c24c0fba3942 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:35.496070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44fdb0b-a0fb-4890-bef0-4bf1f92a70ed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:35.501328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-e44fdb0b-a0fb-4890-bef0-4bf1f92a70ed None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:35.501328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:35.501328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:35.501328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:35.501328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:35.501328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:35.501328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:35.501328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:35.501328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:35.501328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:35.501328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:35.501328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:35.501328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:35.501328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:35.501328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:35.501328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:35.502286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:35.502286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:35.502286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:35.502286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:35.502286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:35.502286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:35.502286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:35.502286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:35.502286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:35.502286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:35.502286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:35.502286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:35.502286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:35.502286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:35.502286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:35.502286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:35.503216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:35.503216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:35.503216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:35.503216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:35.503216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:35.503216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:35.503216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:35.503216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:35.503216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:35.503216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:35.503216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:35.503216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:35.503216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 318/630] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:35.503216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44fdb0b-a0fb-4890-bef0-4bf1f92a70ed None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:35.554710 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:35.598053 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056b4a8e-aa3e-4421-aeac-c24c0fba3942 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:35.608436 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44fdb0b-a0fb-4890-bef0-4bf1f92a70ed None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:35.636427 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e216ca27-d495-470f-8e3b-9097e1136ffa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:35.637158 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e216ca27-d495-470f-8e3b-9097e1136ffa None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:35.637465 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e216ca27-d495-470f-8e3b-9097e1136ffa None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:35.637744 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e216ca27-d495-470f-8e3b-9097e1136ffa None admin] PATH_INFO: `/v3/roles/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:35.643777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a98f0a3-04e1-446d-8fdc-a6644d849f91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:35.644398 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a98f0a3-04e1-446d-8fdc-a6644d849f91 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:35.644674 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a98f0a3-04e1-446d-8fdc-a6644d849f91 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:35.644933 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a98f0a3-04e1-446d-8fdc-a6644d849f91 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:35.646942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:35.647981 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e216ca27-d495-470f-8e3b-9097e1136ffa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:35.653756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-e216ca27-d495-470f-8e3b-9097e1136ffa None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:35.653756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:35.653756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:35.653756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:35.653756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:35.653756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:35.653756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:35.653756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:35.653756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:35.653756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:35.653756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:35.653756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:35.653756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:35.653756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:35.653756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:35.653756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:35.658639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:35.658639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:35.658639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:35.658639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:35.658639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:35.658639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:35.658639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:35.658639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:35.658639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:35.658639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:35.658639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:35.658639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:35.658639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:35.658639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:35.658639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:35.658639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:35.659550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:35.659550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:35.659550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:35.659550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:35.659550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:35.659550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:35.659550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:35.659550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:35.659550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:35.659550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:35.659550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:35.659550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:35.659550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 313/631] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:35.659550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b404c8-cf86-4b9d-afa5-f095459f437e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:35.660313 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b404c8-cf86-4b9d-afa5-f095459f437e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:35.660383 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a98f0a3-04e1-446d-8fdc-a6644d849f91 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:35.660454 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b404c8-cf86-4b9d-afa5-f095459f437e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:35.660521 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a98f0a3-04e1-446d-8fdc-a6644d849f91 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:35.675432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-7a98f0a3-04e1-446d-8fdc-a6644d849f91 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:35.675432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:35.675432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:35.675432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:35.675432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:35.675432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:35.675432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:35.675432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:35.675432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:35.675432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:35.675432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:35.675432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:35.675432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:35.675432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:35.675432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:35.675432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:35.676447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:35.676447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:35.676447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:35.676447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:35.676447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:35.676447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:35.676447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:35.676447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:35.676447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:35.676447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:35.676447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:35.676447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:35.676447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:35.676447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:35.676447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:35.676447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:35.676447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:35.677372 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:35.677372 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:35.677372 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:35.677546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-54b404c8-cf86-4b9d-afa5-f095459f437e None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:35.677632 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 319/632] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:35.677632 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a98f0a3-04e1-446d-8fdc-a6644d849f91 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:35.679485 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-54b404c8-cf86-4b9d-afa5-f095459f437e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:35.747861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 314/633] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:35.755497 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b404c8-cf86-4b9d-afa5-f095459f437e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:35.785198 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a98f0a3-04e1-446d-8fdc-a6644d849f91 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:35.820598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62601b3-cfa2-42a8-a351-0fff17ea2dd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:35.821260 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62601b3-cfa2-42a8-a351-0fff17ea2dd7 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:35.821628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62601b3-cfa2-42a8-a351-0fff17ea2dd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:35.821906 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62601b3-cfa2-42a8-a351-0fff17ea2dd7 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:35.822658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62601b3-cfa2-42a8-a351-0fff17ea2dd7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:35.823030 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62601b3-cfa2-42a8-a351-0fff17ea2dd7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:35.824460 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62601b3-cfa2-42a8-a351-0fff17ea2dd7 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:35.832596 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:35.837757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 320/634] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:27:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:35.837757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62601b3-cfa2-42a8-a351-0fff17ea2dd7 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:35.889981 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:35.906081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b404c8-cf86-4b9d-afa5-f095459f437e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:35.941386 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62601b3-cfa2-42a8-a351-0fff17ea2dd7 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:35.957824 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cf9966-d71e-4201-99c3-32a80ea1ca29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:35.959062 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cf9966-d71e-4201-99c3-32a80ea1ca29 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:35.959524 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cf9966-d71e-4201-99c3-32a80ea1ca29 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:35.959975 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cf9966-d71e-4201-99c3-32a80ea1ca29 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:35.960973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6cf9966-d71e-4201-99c3-32a80ea1ca29 None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:35.961721 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6cf9966-d71e-4201-99c3-32a80ea1ca29 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:35.963316 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6cf9966-d71e-4201-99c3-32a80ea1ca29 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:35.976290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 315/635] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:27:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:35.977566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cf9966-d71e-4201-99c3-32a80ea1ca29 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:35.979365 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6347085-3f4a-4f08-9add-5b3b22e89a26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:35.980047 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6347085-3f4a-4f08-9add-5b3b22e89a26 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:35.980434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6347085-3f4a-4f08-9add-5b3b22e89a26 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:35.980787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6347085-3f4a-4f08-9add-5b3b22e89a26 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:35.981520 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6347085-3f4a-4f08-9add-5b3b22e89a26 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:35.981909 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6347085-3f4a-4f08-9add-5b3b22e89a26 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:35.983405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6347085-3f4a-4f08-9add-5b3b22e89a26 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:35.996007 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 321/636] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:35.996982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6347085-3f4a-4f08-9add-5b3b22e89a26 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:36.091983 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cf9966-d71e-4201-99c3-32a80ea1ca29 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:36.104128 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6347085-3f4a-4f08-9add-5b3b22e89a26 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:36.108467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:36.114431 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:36.126467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03c5f2d-70d7-4726-9296-4bc5ee92d3e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:36.127822 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03c5f2d-70d7-4726-9296-4bc5ee92d3e0 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:36.127822 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03c5f2d-70d7-4726-9296-4bc5ee92d3e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:36.127822 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03c5f2d-70d7-4726-9296-4bc5ee92d3e0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:36.138747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03c5f2d-70d7-4726-9296-4bc5ee92d3e0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:36.140202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03c5f2d-70d7-4726-9296-4bc5ee92d3e0 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:36.144016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875a88dc-8e49-4012-b4f1-c09a1a7b9d1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:36.144840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875a88dc-8e49-4012-b4f1-c09a1a7b9d1f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:36.145323 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875a88dc-8e49-4012-b4f1-c09a1a7b9d1f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:36.145787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875a88dc-8e49-4012-b4f1-c09a1a7b9d1f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:36.155740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-e03c5f2d-70d7-4726-9296-4bc5ee92d3e0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:36.155740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:36.155740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:36.155740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:36.155740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:36.155740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:36.155740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:36.155740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:36.155740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:36.155740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:36.155740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:36.155740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:36.155740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:36.155740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:36.155740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:36.155740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:36.156855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:36.156855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:36.156855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:36.156855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:36.156855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:36.156855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:36.156855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:36.156855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:36.156855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:36.156855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:36.156855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:36.156855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:36.156855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:36.156855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:36.156855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:36.156855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:36.156855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:36.157827 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:36.157827 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:36.157827 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:36.157827 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 316/637] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:36.158067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03c5f2d-70d7-4726-9296-4bc5ee92d3e0 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:36.162727 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875a88dc-8e49-4012-b4f1-c09a1a7b9d1f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:36.164195 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875a88dc-8e49-4012-b4f1-c09a1a7b9d1f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:36.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-875a88dc-8e49-4012-b4f1-c09a1a7b9d1f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:36.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:36.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:36.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:36.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:36.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:36.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:36.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:36.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:36.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:36.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:36.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:36.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:36.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:36.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:36.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:36.185497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:36.185497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:36.185497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:36.185497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:36.185497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:36.185497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:36.185497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:36.185497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:36.185497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:36.185497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:36.185497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:36.185497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:36.185497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:36.185497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:36.185497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:36.185497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:36.185497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:36.186680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:36.186680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:36.186680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:36.193451 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 322/638] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:36.193451 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875a88dc-8e49-4012-b4f1-c09a1a7b9d1f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:36.265624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03c5f2d-70d7-4726-9296-4bc5ee92d3e0 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:36.299631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b66369-47da-43dc-b900-7aa37b42f8db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:36.300250 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b66369-47da-43dc-b900-7aa37b42f8db None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:36.300525 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b66369-47da-43dc-b900-7aa37b42f8db None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:36.300799 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b66369-47da-43dc-b900-7aa37b42f8db None admin] PATH_INFO: `/v3/users/glance` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:36.301085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875a88dc-8e49-4012-b4f1-c09a1a7b9d1f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:36.308916 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b66369-47da-43dc-b900-7aa37b42f8db None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:36.310929 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b66369-47da-43dc-b900-7aa37b42f8db None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:36.322156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-a7b66369-47da-43dc-b900-7aa37b42f8db None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 30 09:27:36.322156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:36.322156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:36.322156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:36.322156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:36.322156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:36.322156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:36.322156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:36.322156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:36.322156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:36.322156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:36.322156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:36.322156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:36.322156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:36.322156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:36.322156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:36.323397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:36.323397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:36.323397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:36.323397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:36.323397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:36.323397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:36.323397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:36.323397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:36.323397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:36.323397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:36.323397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:36.323397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:36.323397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:36.323397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:36.323397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:36.323397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:36.323397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:36.323397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:36.324375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:36.324375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:36.324375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:36.324375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:36.324375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:36.324375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:36.324375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:36.324375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:36.324375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 30 09:27:36.324375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:36.324375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 317/639] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:27:36 2022] GET /identity/v3/users/glance => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:36.325599 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b66369-47da-43dc-b900-7aa37b42f8db None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:36.343264 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bb981e-a4bc-4a33-bf7c-b3a7743c921f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:36.343829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bb981e-a4bc-4a33-bf7c-b3a7743c921f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:36.344112 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bb981e-a4bc-4a33-bf7c-b3a7743c921f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:36.344384 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bb981e-a4bc-4a33-bf7c-b3a7743c921f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:36.355420 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bb981e-a4bc-4a33-bf7c-b3a7743c921f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:36.355568 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:36.360736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-b7bb981e-a4bc-4a33-bf7c-b3a7743c921f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:36.360736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:36.360736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:36.360736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:36.360736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:36.360736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:36.360736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:36.360736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:36.360736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:36.360736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:36.360736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:36.360736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:36.360736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:36.360736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:36.360736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:36.360736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:36.361938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:36.361938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:36.361938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:36.361938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:36.361938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:36.361938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:36.361938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:36.361938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:36.361938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:36.361938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:36.361938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:36.361938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:36.361938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:36.361938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:36.361938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:36.361938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:36.362850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:36.362850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:36.362850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:36.362850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:36.362850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:36.362850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:36.362850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:36.362850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:36.362850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:36.362850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:36.362850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:36.362850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:36.362850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 323/640] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:36 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:36.362850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bb981e-a4bc-4a33-bf7c-b3a7743c921f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:36.391717 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:36.440721 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b66369-47da-43dc-b900-7aa37b42f8db None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:36.472126 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bb981e-a4bc-4a33-bf7c-b3a7743c921f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:36.476419 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8d9cf4-7753-4c1c-b455-b77f26f1d989 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:36.477102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8d9cf4-7753-4c1c-b455-b77f26f1d989 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:36.477453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8d9cf4-7753-4c1c-b455-b77f26f1d989 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:36.477797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8d9cf4-7753-4c1c-b455-b77f26f1d989 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:36.478502 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8d9cf4-7753-4c1c-b455-b77f26f1d989 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:36.478897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8d9cf4-7753-4c1c-b455-b77f26f1d989 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:36.480435 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8d9cf4-7753-4c1c-b455-b77f26f1d989 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:36.497336 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 318/641] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:36.497336 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8d9cf4-7753-4c1c-b455-b77f26f1d989 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:36.510956 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84e46b8-c874-4480-ad27-fbf323e5a52f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:36.511632 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84e46b8-c874-4480-ad27-fbf323e5a52f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:36.511985 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84e46b8-c874-4480-ad27-fbf323e5a52f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:36.512346 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84e46b8-c874-4480-ad27-fbf323e5a52f None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:36.513626 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84e46b8-c874-4480-ad27-fbf323e5a52f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:36.513929 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84e46b8-c874-4480-ad27-fbf323e5a52f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:36.515358 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84e46b8-c874-4480-ad27-fbf323e5a52f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:36.526403 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 324/642] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:36.527310 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84e46b8-c874-4480-ad27-fbf323e5a52f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:36.594616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:36.607372 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8d9cf4-7753-4c1c-b455-b77f26f1d989 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:36.615526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:36.636684 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84e46b8-c874-4480-ad27-fbf323e5a52f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:36.644755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9c53be-56b7-4a11-aa01-1a88433a9996 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:36.645357 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9c53be-56b7-4a11-aa01-1a88433a9996 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:36.645634 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9c53be-56b7-4a11-aa01-1a88433a9996 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:36.645902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9c53be-56b7-4a11-aa01-1a88433a9996 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:36.646647 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9c53be-56b7-4a11-aa01-1a88433a9996 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:36.646950 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9c53be-56b7-4a11-aa01-1a88433a9996 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:36.648376 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9c53be-56b7-4a11-aa01-1a88433a9996 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:36.673657 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04071a0-96f5-4871-a3be-3d73f3d36817 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:36.673881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 319/643] 10.222.0.39 () {56 vars in 1137 bytes} [Fri Sep 30 09:27:36 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:36.674402 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04071a0-96f5-4871-a3be-3d73f3d36817 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:36.674778 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9c53be-56b7-4a11-aa01-1a88433a9996 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:36.674884 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04071a0-96f5-4871-a3be-3d73f3d36817 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:36.675172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04071a0-96f5-4871-a3be-3d73f3d36817 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:36.675876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04071a0-96f5-4871-a3be-3d73f3d36817 None admin] RBAC: Adding query filter params (name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:36.676241 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04071a0-96f5-4871-a3be-3d73f3d36817 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:36.677885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04071a0-96f5-4871-a3be-3d73f3d36817 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:36.687867 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 325/644] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:27:36 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:36.688675 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04071a0-96f5-4871-a3be-3d73f3d36817 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:36.787021 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9c53be-56b7-4a11-aa01-1a88433a9996 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:36.796178 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04071a0-96f5-4871-a3be-3d73f3d36817 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:36.824762 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82eb51e3-8bfa-4713-8b11-b6ef8e9d573f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:36.825724 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82eb51e3-8bfa-4713-8b11-b6ef8e9d573f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:36.826203 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82eb51e3-8bfa-4713-8b11-b6ef8e9d573f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:36.826484 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82eb51e3-8bfa-4713-8b11-b6ef8e9d573f None admin] PATH_INFO: `/v3/users/nova` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:36.831167 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250440a2-a6c3-4eca-a543-10d1cb1e7d08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:36.831797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250440a2-a6c3-4eca-a543-10d1cb1e7d08 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:36.832076 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250440a2-a6c3-4eca-a543-10d1cb1e7d08 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:36.832348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250440a2-a6c3-4eca-a543-10d1cb1e7d08 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:36.833577 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82eb51e3-8bfa-4713-8b11-b6ef8e9d573f None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:36.835054 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82eb51e3-8bfa-4713-8b11-b6ef8e9d573f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:36.840972 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250440a2-a6c3-4eca-a543-10d1cb1e7d08 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:36.842646 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250440a2-a6c3-4eca-a543-10d1cb1e7d08 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:36.846758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-82eb51e3-8bfa-4713-8b11-b6ef8e9d573f None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 30 09:27:36.846758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:36.846758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:36.846758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:36.846758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:36.846758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:36.846758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:36.846758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:36.846758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:36.846758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:36.846758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:36.846758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:36.846758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:36.846758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:36.846758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:36.846758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:36.847688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:36.847688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:36.847688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:36.847688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:36.847688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:36.847688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:36.847688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:36.847688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:36.847688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:36.847688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:36.847688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:36.847688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:36.847688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:36.847688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:36.847688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:36.847688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:36.847688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:36.847688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:36.848806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:36.848806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:36.848806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:36.848806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:36.848806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:36.848806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:36.848806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:36.848806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:36.848806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 30 09:27:36.848806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:36.848806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 320/645] 10.222.0.39 () {56 vars in 1093 bytes} [Fri Sep 30 09:27:36 2022] GET /identity/v3/users/nova => generated 82 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:36.848806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82eb51e3-8bfa-4713-8b11-b6ef8e9d573f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:36.858321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-250440a2-a6c3-4eca-a543-10d1cb1e7d08 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 30 09:27:36.858321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:36.858321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:36.858321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:36.858321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:36.858321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:36.858321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:36.858321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:36.858321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:36.858321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:36.858321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:36.858321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:36.858321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:36.858321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:36.858321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:36.858321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:36.859320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:36.859320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:36.859320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:36.859320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:36.859320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:36.859320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:36.859320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:36.859320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:36.859320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:36.859320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:36.859320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:36.859320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:36.859320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:36.859320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:36.859320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:36.859320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:36.859320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:36.859320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:36.860322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:36.860322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:36.860322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:36.860322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:36.860322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:36.860322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:36.860322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:36.860322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:36.860322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 30 09:27:36.860322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:36.866358 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 326/646] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:27:36 2022] GET /identity/v3/users/cinder => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:36.866521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:36.866653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250440a2-a6c3-4eca-a543-10d1cb1e7d08 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:36.866653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:36.962809 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82eb51e3-8bfa-4713-8b11-b6ef8e9d573f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:36.966263 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250440a2-a6c3-4eca-a543-10d1cb1e7d08 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:36.997853 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20f6016-e06b-41de-8573-925ff642a3ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:36.998461 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20f6016-e06b-41de-8573-925ff642a3ca None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:36.998779 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20f6016-e06b-41de-8573-925ff642a3ca None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:36.999057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20f6016-e06b-41de-8573-925ff642a3ca None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:37.003083 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30783b25-5cc3-4528-96fd-67184869bbab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:37.003963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30783b25-5cc3-4528-96fd-67184869bbab None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:37.004515 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30783b25-5cc3-4528-96fd-67184869bbab None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:37.005006 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30783b25-5cc3-4528-96fd-67184869bbab None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:37.016561 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30783b25-5cc3-4528-96fd-67184869bbab None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:37.018407 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30783b25-5cc3-4528-96fd-67184869bbab None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:37.034180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-30783b25-5cc3-4528-96fd-67184869bbab None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:37.034180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:37.034180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:37.034180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:37.034180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:37.034180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:37.034180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:37.034180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:37.034180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:37.034180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:37.034180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:37.034180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:37.034180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:37.034180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:37.034180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:37.034180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:37.035268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:37.035268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:37.035268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:37.035268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:37.035268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:37.035268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:37.035268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:37.035268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:37.035268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:37.035268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:37.035268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:37.035268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:37.035268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:37.035268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:37.035268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:37.035268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:37.035268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:37.036360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:37.036360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:37.036360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:37.038135 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 327/647] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:37.039146 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30783b25-5cc3-4528-96fd-67184869bbab None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:37.076282 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20f6016-e06b-41de-8573-925ff642a3ca None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:37.078880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20f6016-e06b-41de-8573-925ff642a3ca None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:37.088251 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:37.098662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:37.155985 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30783b25-5cc3-4528-96fd-67184869bbab None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:37.192237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b801f0-39c4-4a48-828b-40cd7a3fef5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:37.192879 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b801f0-39c4-4a48-828b-40cd7a3fef5c None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:37.193277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b801f0-39c4-4a48-828b-40cd7a3fef5c None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:37.194917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b801f0-39c4-4a48-828b-40cd7a3fef5c None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:37.194917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b801f0-39c4-4a48-828b-40cd7a3fef5c None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:37.194917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b801f0-39c4-4a48-828b-40cd7a3fef5c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:37.196981 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b801f0-39c4-4a48-828b-40cd7a3fef5c None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:37.204517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 321/648] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:27:37.205401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20f6016-e06b-41de-8573-925ff642a3ca None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:37.214177 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 328/649] 10.222.0.39 () {56 vars in 1133 bytes} [Fri Sep 30 09:27:37 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:37.215130 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b801f0-39c4-4a48-828b-40cd7a3fef5c None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:37.307572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20f6016-e06b-41de-8573-925ff642a3ca None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:37.313412 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b801f0-39c4-4a48-828b-40cd7a3fef5c None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:37.314781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:37.327090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:37.341166 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48dd4b16-b83b-44e8-b674-72709876aa94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:37.341768 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dd4b16-b83b-44e8-b674-72709876aa94 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:37.342048 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dd4b16-b83b-44e8-b674-72709876aa94 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:37.342322 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dd4b16-b83b-44e8-b674-72709876aa94 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:37.343172 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48dd4b16-b83b-44e8-b674-72709876aa94 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:37.343738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48dd4b16-b83b-44e8-b674-72709876aa94 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:37.345483 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48dd4b16-b83b-44e8-b674-72709876aa94 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:37.353369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d5a16f-866d-46f8-bea6-df2413926f54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:37.353369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d5a16f-866d-46f8-bea6-df2413926f54 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:37.353369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d5a16f-866d-46f8-bea6-df2413926f54 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:37.353369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d5a16f-866d-46f8-bea6-df2413926f54 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:37.353726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d5a16f-866d-46f8-bea6-df2413926f54 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:37.353726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d5a16f-866d-46f8-bea6-df2413926f54 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:37.353726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d5a16f-866d-46f8-bea6-df2413926f54 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:37.362931 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 322/650] 10.222.0.39 () {56 vars in 1137 bytes} [Fri Sep 30 09:27:37 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:37.368461 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 329/651] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:37.368608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48dd4b16-b83b-44e8-b674-72709876aa94 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:37.368696 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d5a16f-866d-46f8-bea6-df2413926f54 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:37.461846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d5a16f-866d-46f8-bea6-df2413926f54 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:37.464799 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48dd4b16-b83b-44e8-b674-72709876aa94 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:37.495261 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508aa462-7812-47b1-9283-d845c48f50e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:37.495860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508aa462-7812-47b1-9283-d845c48f50e6 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:37.496144 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508aa462-7812-47b1-9283-d845c48f50e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:37.496420 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508aa462-7812-47b1-9283-d845c48f50e6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:37.497762 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2540c3ec-18b9-4257-b7a9-82f918dead7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:37.498385 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2540c3ec-18b9-4257-b7a9-82f918dead7b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:37.498706 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2540c3ec-18b9-4257-b7a9-82f918dead7b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:37.498985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2540c3ec-18b9-4257-b7a9-82f918dead7b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:37.509218 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2540c3ec-18b9-4257-b7a9-82f918dead7b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:37.510684 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2540c3ec-18b9-4257-b7a9-82f918dead7b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:37.525426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-2540c3ec-18b9-4257-b7a9-82f918dead7b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:37.525426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:37.525426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:37.525426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:37.525426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:37.525426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:37.525426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:37.525426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:37.525426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:37.525426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:37.525426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:37.525426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:37.525426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:37.525426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:37.525426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:37.525426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:37.526401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:37.526401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:37.526401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:37.526401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:37.526401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:37.526401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:37.526401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:37.526401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:37.526401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:37.526401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:37.526401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:37.526401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:37.526401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:37.526401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:37.526401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:37.526401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:37.526401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:37.527902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:37.527902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:37.527902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:37.527902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 323/652] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:37.527902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89be55c3-1a7b-444f-8024-93ae282e7582 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:37.528830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89be55c3-1a7b-444f-8024-93ae282e7582 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:37.528937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89be55c3-1a7b-444f-8024-93ae282e7582 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:37.530163 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 324/653] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:37.533522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89be55c3-1a7b-444f-8024-93ae282e7582 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:37.545095 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:37.550903 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:37.572223 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508aa462-7812-47b1-9283-d845c48f50e6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:37.573773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508aa462-7812-47b1-9283-d845c48f50e6 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:37.635305 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89be55c3-1a7b-444f-8024-93ae282e7582 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:37.667945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2838e056-580e-41f6-847b-ed8bb36db1f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:37.668539 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2838e056-580e-41f6-847b-ed8bb36db1f7 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:37.668814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2838e056-580e-41f6-847b-ed8bb36db1f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:37.669085 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2838e056-580e-41f6-847b-ed8bb36db1f7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:37.669610 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 330/654] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:27:37.670277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508aa462-7812-47b1-9283-d845c48f50e6 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:37.735439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2838e056-580e-41f6-847b-ed8bb36db1f7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:37.736895 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2838e056-580e-41f6-847b-ed8bb36db1f7 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:37.767882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508aa462-7812-47b1-9283-d845c48f50e6 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:37.799696 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2412ab12-85ef-4c97-8235-fbc478eb1419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:37.800323 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2412ab12-85ef-4c97-8235-fbc478eb1419 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:37.800603 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2412ab12-85ef-4c97-8235-fbc478eb1419 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:37.800877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2412ab12-85ef-4c97-8235-fbc478eb1419 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:37.807917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2412ab12-85ef-4c97-8235-fbc478eb1419 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:37.809884 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2412ab12-85ef-4c97-8235-fbc478eb1419 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:37.821477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-2412ab12-85ef-4c97-8235-fbc478eb1419 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 30 09:27:37.821477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:37.821477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:37.821477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:37.821477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:37.821477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:37.821477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:37.821477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:37.821477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:37.821477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:37.821477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:37.821477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:37.821477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:37.821477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:37.821477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:37.821477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:37.822779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:37.822779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:37.822779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:37.822779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:37.822779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:37.822779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:37.822779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:37.822779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:37.822779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:37.822779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:37.822779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:37.822779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:37.822779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:37.822779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:37.822779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:37.822779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:37.822779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:37.822779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:37.824059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:37.824059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:37.824059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:37.824059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:37.824059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:37.824059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:37.824059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:37.824059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:37.824059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 30 09:27:37.824059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:37.824059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 331/655] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:37 2022] GET /identity/v3/users/neutron => generated 85 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:37.824059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2412ab12-85ef-4c97-8235-fbc478eb1419 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:37.834346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 325/656] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:27:37.835612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cfcd98-7206-4556-979c-62a9462e19de None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:37.835900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cfcd98-7206-4556-979c-62a9462e19de None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:37.836177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cfcd98-7206-4556-979c-62a9462e19de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:37.853583 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-40cfcd98-7206-4556-979c-62a9462e19de None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:37.854811 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-40cfcd98-7206-4556-979c-62a9462e19de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:37.867509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:37.895875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:37.920062 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 326/657] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:37.920871 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cfcd98-7206-4556-979c-62a9462e19de None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:37.923940 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2412ab12-85ef-4c97-8235-fbc478eb1419 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:37.955821 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:37.960740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead482ca-c5e0-4a6e-9921-9f597d8b5e3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:37.961491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead482ca-c5e0-4a6e-9921-9f597d8b5e3a None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:37.961740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead482ca-c5e0-4a6e-9921-9f597d8b5e3a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:37.962014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead482ca-c5e0-4a6e-9921-9f597d8b5e3a None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:37.962687 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead482ca-c5e0-4a6e-9921-9f597d8b5e3a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:37.965624 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead482ca-c5e0-4a6e-9921-9f597d8b5e3a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:37.965624 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead482ca-c5e0-4a6e-9921-9f597d8b5e3a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:37.969246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:37.981930 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 332/658] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:37.981930 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead482ca-c5e0-4a6e-9921-9f597d8b5e3a None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:38.027050 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cfcd98-7206-4556-979c-62a9462e19de None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:38.064004 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a306e538-ad4d-4026-8315-dc43fc617811 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:38.064645 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a306e538-ad4d-4026-8315-dc43fc617811 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:38.064931 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a306e538-ad4d-4026-8315-dc43fc617811 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:38.065211 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a306e538-ad4d-4026-8315-dc43fc617811 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:38.078186 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a306e538-ad4d-4026-8315-dc43fc617811 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:38.079762 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a306e538-ad4d-4026-8315-dc43fc617811 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:38.091970 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead482ca-c5e0-4a6e-9921-9f597d8b5e3a None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:38.094106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-a306e538-ad4d-4026-8315-dc43fc617811 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:38.094106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:38.094106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:38.094106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:38.094106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:38.094106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:38.094106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:38.094106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:38.094106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:38.094106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:38.094106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:38.094106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:38.094106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:38.094106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:38.094106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:38.094106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:38.095191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:38.095191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:38.095191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:38.095191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:38.095191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:38.095191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:38.095191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:38.095191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:38.095191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:38.095191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:38.095191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:38.095191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:38.095191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:38.095191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:38.095191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:38.095191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:38.095191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:38.096160 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:38.096160 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:38.096160 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:38.096160 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 327/659] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:38.096430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a306e538-ad4d-4026-8315-dc43fc617811 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:38.134324 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaba7b42-e3f8-4a1c-a40f-ae71e8fc500d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:38.135118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaba7b42-e3f8-4a1c-a40f-ae71e8fc500d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:38.135407 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaba7b42-e3f8-4a1c-a40f-ae71e8fc500d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:38.136414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaba7b42-e3f8-4a1c-a40f-ae71e8fc500d None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:38.137147 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaba7b42-e3f8-4a1c-a40f-ae71e8fc500d None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:38.137490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaba7b42-e3f8-4a1c-a40f-ae71e8fc500d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:38.138959 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaba7b42-e3f8-4a1c-a40f-ae71e8fc500d None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:38.162469 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 333/660] 10.222.0.39 () {56 vars in 1139 bytes} [Fri Sep 30 09:27:37 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:38.162469 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaba7b42-e3f8-4a1c-a40f-ae71e8fc500d None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:38.207827 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:38.211988 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a306e538-ad4d-4026-8315-dc43fc617811 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:38.213060 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:38.246190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffa0843-26c3-4e8a-85e2-1c3779add73f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:38.247406 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffa0843-26c3-4e8a-85e2-1c3779add73f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:38.247685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffa0843-26c3-4e8a-85e2-1c3779add73f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:38.247961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffa0843-26c3-4e8a-85e2-1c3779add73f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:38.258079 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffa0843-26c3-4e8a-85e2-1c3779add73f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:38.259656 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffa0843-26c3-4e8a-85e2-1c3779add73f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:38.261987 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaba7b42-e3f8-4a1c-a40f-ae71e8fc500d None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:38.274182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-2ffa0843-26c3-4e8a-85e2-1c3779add73f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:38.274182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:38.274182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:38.274182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:38.274182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:38.274182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:38.274182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:38.274182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:38.274182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:38.274182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:38.274182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:38.274182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:38.274182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:38.274182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:38.274182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:38.274182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:38.275373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:38.275373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:38.275373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:38.275373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:38.275373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:38.275373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:38.275373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:38.275373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:38.275373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:38.275373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:38.275373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:38.275373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:38.275373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:38.275373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:38.275373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:38.275373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:38.275373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:38.276485 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:38.276485 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:38.276485 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:38.276485 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 328/661] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:38.276485 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffa0843-26c3-4e8a-85e2-1c3779add73f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:38.297643 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d59c9b2-cb0d-4dd9-aa18-538b0b0222ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:38.298270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d59c9b2-cb0d-4dd9-aa18-538b0b0222ee None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:38.298548 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d59c9b2-cb0d-4dd9-aa18-538b0b0222ee None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:38.298849 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d59c9b2-cb0d-4dd9-aa18-538b0b0222ee None admin] PATH_INFO: `/v3/projects/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:38.305538 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d59c9b2-cb0d-4dd9-aa18-538b0b0222ee None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:38.307122 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d59c9b2-cb0d-4dd9-aa18-538b0b0222ee None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:38.319334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-9d59c9b2-cb0d-4dd9-aa18-538b0b0222ee None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:38.319334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:38.319334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:38.319334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:38.319334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:38.319334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:38.319334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:38.319334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:38.319334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:38.319334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:38.319334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:38.319334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:38.319334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:38.319334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:38.319334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:38.320279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:38.320279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:38.320279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:38.320279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:38.320279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:38.320279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:38.320279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:38.320279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:38.320279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:38.320279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:38.320279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:38.320279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:38.320279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:38.320279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:38.320279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:38.320279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:38.320279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:38.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:38.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:38.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:38.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:38.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:38.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:38.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:38.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:38.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 334/662] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:38.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d59c9b2-cb0d-4dd9-aa18-538b0b0222ee None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:38.377998 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffa0843-26c3-4e8a-85e2-1c3779add73f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:38.411834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6596d66-3f7e-414f-ab86-288279610022 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:38.412438 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6596d66-3f7e-414f-ab86-288279610022 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:38.412771 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6596d66-3f7e-414f-ab86-288279610022 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:38.412999 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6596d66-3f7e-414f-ab86-288279610022 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:38.413632 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6596d66-3f7e-414f-ab86-288279610022 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:38.413936 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6596d66-3f7e-414f-ab86-288279610022 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:38.415341 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6596d66-3f7e-414f-ab86-288279610022 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:38.421223 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d59c9b2-cb0d-4dd9-aa18-538b0b0222ee None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:38.427269 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 329/663] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:38.428057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6596d66-3f7e-414f-ab86-288279610022 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:38.432603 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:38.440344 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:38.454954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1944679d-d1bc-465d-ad80-4fb58c3fd820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:38.455558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1944679d-d1bc-465d-ad80-4fb58c3fd820 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:38.455841 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1944679d-d1bc-465d-ad80-4fb58c3fd820 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:38.456115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1944679d-d1bc-465d-ad80-4fb58c3fd820 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:38.522191 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1944679d-d1bc-465d-ad80-4fb58c3fd820 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:38.523448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6596d66-3f7e-414f-ab86-288279610022 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:38.523691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1944679d-d1bc-465d-ad80-4fb58c3fd820 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:38.557038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a60e457-495f-47ab-bd75-ed26e35056cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:38.557661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a60e457-495f-47ab-bd75-ed26e35056cd None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:38.557944 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a60e457-495f-47ab-bd75-ed26e35056cd None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:38.558220 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a60e457-495f-47ab-bd75-ed26e35056cd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:38.568212 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a60e457-495f-47ab-bd75-ed26e35056cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:38.573072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-4a60e457-495f-47ab-bd75-ed26e35056cd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:38.573072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:38.573072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:38.573072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:38.573072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:38.573072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:38.573072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:38.573072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:38.573072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:38.573072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:38.573072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:38.573072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:38.573072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:38.573072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:38.573072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:38.573072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:38.574204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:38.574204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:38.574204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:38.574204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:38.574204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:38.574204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:38.574204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:38.574204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:38.574204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:38.574204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:38.574204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:38.574204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:38.574204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:38.574204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:38.574204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:38.574204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:38.575185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:38.575185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:38.575185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:38.575185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:38.575185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:38.575185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:38.575185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:38.575185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:38.575185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:38.575185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:38.575185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:38.575185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:38.575185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 330/664] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:38 2022] GET /identity/v3/roles/service => generated 85 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:38.575185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a60e457-495f-47ab-bd75-ed26e35056cd None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:38.622670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 335/665] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:27:38.623501 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1944679d-d1bc-465d-ad80-4fb58c3fd820 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:38.634815 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:38.647797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:38.677874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a60e457-495f-47ab-bd75-ed26e35056cd None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:38.710254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb43fe7-3b53-4775-90cb-3124c7782034 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:38.710885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb43fe7-3b53-4775-90cb-3124c7782034 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:38.711169 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb43fe7-3b53-4775-90cb-3124c7782034 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:38.711444 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb43fe7-3b53-4775-90cb-3124c7782034 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:38.712077 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb43fe7-3b53-4775-90cb-3124c7782034 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:38.712386 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb43fe7-3b53-4775-90cb-3124c7782034 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:38.713744 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb43fe7-3b53-4775-90cb-3124c7782034 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:38.721276 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1944679d-d1bc-465d-ad80-4fb58c3fd820 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:38.725664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 331/666] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:38.726447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb43fe7-3b53-4775-90cb-3124c7782034 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:38.755212 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496f4448-c8e2-483e-aefe-e2f38a3a8370 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:38.755813 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496f4448-c8e2-483e-aefe-e2f38a3a8370 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:38.756100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496f4448-c8e2-483e-aefe-e2f38a3a8370 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:38.756631 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496f4448-c8e2-483e-aefe-e2f38a3a8370 None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:38.757311 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496f4448-c8e2-483e-aefe-e2f38a3a8370 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:38.757625 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496f4448-c8e2-483e-aefe-e2f38a3a8370 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:38.759105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496f4448-c8e2-483e-aefe-e2f38a3a8370 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:38.770457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 336/667] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:38.771271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496f4448-c8e2-483e-aefe-e2f38a3a8370 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:38.826024 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb43fe7-3b53-4775-90cb-3124c7782034 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:38.838023 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:38.855544 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:38.861094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24722b8c-f72c-4823-9276-b22688da060f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:38.861736 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24722b8c-f72c-4823-9276-b22688da060f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:38.862009 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24722b8c-f72c-4823-9276-b22688da060f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:38.862276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24722b8c-f72c-4823-9276-b22688da060f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:38.868106 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496f4448-c8e2-483e-aefe-e2f38a3a8370 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:38.868815 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24722b8c-f72c-4823-9276-b22688da060f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:38.870248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24722b8c-f72c-4823-9276-b22688da060f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:38.881114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-24722b8c-f72c-4823-9276-b22688da060f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:38.881114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:38.881114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:38.881114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:38.881114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:38.881114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:38.881114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:38.881114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:38.881114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:38.881114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:38.881114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:38.881114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:38.881114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:38.881114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:38.881114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:38.888468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:38.888468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:38.888468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:38.888468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:38.888468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:38.888468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:38.888468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:38.888468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:38.888468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:38.888468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:38.888468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:38.888468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:38.888468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:38.888468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:38.888468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:38.888468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:38.888468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:38.889579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:38.889579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:38.889579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:38.889579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:38.889579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:38.889579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:38.889579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:38.889579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:38.889579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 332/668] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:38.889579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24722b8c-f72c-4823-9276-b22688da060f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:38.907461 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388ecb5d-4429-4797-850d-562db9514b24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:38.908104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388ecb5d-4429-4797-850d-562db9514b24 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:38.908338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388ecb5d-4429-4797-850d-562db9514b24 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:38.908614 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388ecb5d-4429-4797-850d-562db9514b24 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:38.909272 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388ecb5d-4429-4797-850d-562db9514b24 None admin] RBAC: Adding query filter params (name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:38.909579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388ecb5d-4429-4797-850d-562db9514b24 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:38.910982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388ecb5d-4429-4797-850d-562db9514b24 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:38.918765 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 337/669] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:27:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:38.919562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388ecb5d-4429-4797-850d-562db9514b24 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:38.981800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24722b8c-f72c-4823-9276-b22688da060f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:39.015000 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284e4dc8-6da6-43bb-84d2-4d68d2d223e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:39.015621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284e4dc8-6da6-43bb-84d2-4d68d2d223e3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:39.015907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284e4dc8-6da6-43bb-84d2-4d68d2d223e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:39.016178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284e4dc8-6da6-43bb-84d2-4d68d2d223e3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:39.016535 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388ecb5d-4429-4797-850d-562db9514b24 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:39.026700 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284e4dc8-6da6-43bb-84d2-4d68d2d223e3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:39.028530 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284e4dc8-6da6-43bb-84d2-4d68d2d223e3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:39.042640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-284e4dc8-6da6-43bb-84d2-4d68d2d223e3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:39.042640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:39.042640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:39.042640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:39.042640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:39.042640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:39.042640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:39.042640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:39.042640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:39.042640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:39.042640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:39.042640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:39.042640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:39.042640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:39.042640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:39.042640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:39.043677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:39.043677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:39.043677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:39.043677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:39.043677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:39.043677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:39.043677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:39.043677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:39.043677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:39.043677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:39.043677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:39.043677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:39.043677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:39.043677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:39.043677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:39.043677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:39.043677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:39.044591 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:39.044591 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:39.044591 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:39.044591 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 333/670] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:39.044591 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284e4dc8-6da6-43bb-84d2-4d68d2d223e3 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:39.049388 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:39.054578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2e5e79-940a-4b0b-89f1-004a0dfea05f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:39.055247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2e5e79-940a-4b0b-89f1-004a0dfea05f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:39.055529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2e5e79-940a-4b0b-89f1-004a0dfea05f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:39.055803 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2e5e79-940a-4b0b-89f1-004a0dfea05f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:39.062702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2e5e79-940a-4b0b-89f1-004a0dfea05f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:39.064146 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2e5e79-940a-4b0b-89f1-004a0dfea05f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:39.075937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-5f2e5e79-940a-4b0b-89f1-004a0dfea05f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:39.075937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:39.075937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:39.075937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:39.075937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:39.075937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:39.075937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:39.075937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:39.075937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:39.075937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:39.075937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:39.075937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:39.075937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:39.075937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:39.075937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:39.076850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:39.076850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:39.076850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:39.076850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:39.076850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:39.076850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:39.076850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:39.076850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:39.076850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:39.076850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:39.076850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:39.076850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:39.076850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:39.076850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:39.076850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:39.076850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:39.076850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:39.077854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:39.077854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:39.077854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:39.077854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:39.077854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:39.077854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:39.077854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:39.077854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:39.077854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 338/671] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:39.077854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2e5e79-940a-4b0b-89f1-004a0dfea05f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:39.088619 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:39.145402 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284e4dc8-6da6-43bb-84d2-4d68d2d223e3 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:39.179019 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b41264b-6ee4-44b1-817b-c94811a1f350 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:39.179624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b41264b-6ee4-44b1-817b-c94811a1f350 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:39.179757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2e5e79-940a-4b0b-89f1-004a0dfea05f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:39.179899 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b41264b-6ee4-44b1-817b-c94811a1f350 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:39.180170 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b41264b-6ee4-44b1-817b-c94811a1f350 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:39.180876 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b41264b-6ee4-44b1-817b-c94811a1f350 None admin] RBAC: Adding query filter params (role.id=144dd855f74945548ea65db37acda1c1, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=bf907d4033064fb8a2707b1bbb0a180b) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:39.181183 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b41264b-6ee4-44b1-817b-c94811a1f350 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:39.182615 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b41264b-6ee4-44b1-817b-c94811a1f350 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:39.188730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 334/672] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:27:39 2022] GET /identity/v3/role_assignments?user.id=bf907d4033064fb8a2707b1bbb0a180b&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=144dd855f74945548ea65db37acda1c1 => generated 608 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:39.189489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b41264b-6ee4-44b1-817b-c94811a1f350 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:39.230238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4524bc7c-421f-47fb-9855-16e2f2a1f23c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:39.231239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4524bc7c-421f-47fb-9855-16e2f2a1f23c None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:39.231688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4524bc7c-421f-47fb-9855-16e2f2a1f23c None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:39.232179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4524bc7c-421f-47fb-9855-16e2f2a1f23c None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:39.233596 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4524bc7c-421f-47fb-9855-16e2f2a1f23c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:39.234074 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4524bc7c-421f-47fb-9855-16e2f2a1f23c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:39.236466 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4524bc7c-421f-47fb-9855-16e2f2a1f23c None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:39.254877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 339/673] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:39.256164 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4524bc7c-421f-47fb-9855-16e2f2a1f23c None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:39.302487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b41264b-6ee4-44b1-817b-c94811a1f350 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:39.312498 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:39.317258 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:39.342315 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d07955-67dd-4447-a799-3608f2995cea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:39.343014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d07955-67dd-4447-a799-3608f2995cea None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:39.343298 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d07955-67dd-4447-a799-3608f2995cea None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:39.343575 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d07955-67dd-4447-a799-3608f2995cea None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:39.355719 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d07955-67dd-4447-a799-3608f2995cea None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:39.356708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d07955-67dd-4447-a799-3608f2995cea None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:39.373150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-a9d07955-67dd-4447-a799-3608f2995cea None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:39.373150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:39.373150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:39.373150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:39.373150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:39.373150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:39.373150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:39.373150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:39.373150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:39.373150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:39.373150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:39.373150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:39.373150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:39.373150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:39.373150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:39.373150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:39.378718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:39.378718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:39.378718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:39.378718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:39.378718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:39.378718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:39.378718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:39.378718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:39.378718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:39.378718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:39.378718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:39.378718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:39.378718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:39.378718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:39.378718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:39.378718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:39.378718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:39.379784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:39.379784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:39.379784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:39.379956 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4524bc7c-421f-47fb-9855-16e2f2a1f23c None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:39.380038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 335/674] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:39.380038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d07955-67dd-4447-a799-3608f2995cea None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:39.414127 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d46327e-f092-48ac-92ea-8560a0255796 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:39.414781 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d46327e-f092-48ac-92ea-8560a0255796 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:39.415111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d46327e-f092-48ac-92ea-8560a0255796 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:39.415400 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d46327e-f092-48ac-92ea-8560a0255796 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:39.416045 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d46327e-f092-48ac-92ea-8560a0255796 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:39.416401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d46327e-f092-48ac-92ea-8560a0255796 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:39.417842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d46327e-f092-48ac-92ea-8560a0255796 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:39.431439 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 340/675] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:39.432264 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d46327e-f092-48ac-92ea-8560a0255796 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:39.491834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d07955-67dd-4447-a799-3608f2995cea None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:39.531484 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9435483b-a4bd-4b69-8422-f3eb420503ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:39.532097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9435483b-a4bd-4b69-8422-f3eb420503ac None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:39.532375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9435483b-a4bd-4b69-8422-f3eb420503ac None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:39.532647 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9435483b-a4bd-4b69-8422-f3eb420503ac None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:39.533330 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9435483b-a4bd-4b69-8422-f3eb420503ac None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:39.533638 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9435483b-a4bd-4b69-8422-f3eb420503ac None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:39.535146 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9435483b-a4bd-4b69-8422-f3eb420503ac None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:39.539640 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d46327e-f092-48ac-92ea-8560a0255796 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:39.547732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 336/676] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:39.548507 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9435483b-a4bd-4b69-8422-f3eb420503ac None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:39.558356 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:39.568326 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:39.581001 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e43ab95-cc14-4702-989f-032e60380d2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:39.581650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e43ab95-cc14-4702-989f-032e60380d2f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:39.581931 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e43ab95-cc14-4702-989f-032e60380d2f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:39.582213 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e43ab95-cc14-4702-989f-032e60380d2f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:39.582982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e43ab95-cc14-4702-989f-032e60380d2f None admin] RBAC: Adding query filter params (role.id=144dd855f74945548ea65db37acda1c1, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=7ff68fb87700423ab53c90520a6bd3b9) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:39.583287 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e43ab95-cc14-4702-989f-032e60380d2f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:39.584744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e43ab95-cc14-4702-989f-032e60380d2f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:39.590716 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 341/677] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:27:39 2022] GET /identity/v3/role_assignments?user.id=7ff68fb87700423ab53c90520a6bd3b9&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=144dd855f74945548ea65db37acda1c1 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:39.591523 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e43ab95-cc14-4702-989f-032e60380d2f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:39.677841 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9435483b-a4bd-4b69-8422-f3eb420503ac None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:39.727926 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab43553-5024-48cc-993d-f9130d346bcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:39.728820 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab43553-5024-48cc-993d-f9130d346bcb None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:39.729115 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab43553-5024-48cc-993d-f9130d346bcb None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:39.729391 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab43553-5024-48cc-993d-f9130d346bcb None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:39.730046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab43553-5024-48cc-993d-f9130d346bcb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:39.730349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab43553-5024-48cc-993d-f9130d346bcb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:39.732315 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab43553-5024-48cc-993d-f9130d346bcb None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:39.737884 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e43ab95-cc14-4702-989f-032e60380d2f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:39.746377 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 337/678] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:39.747887 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab43553-5024-48cc-993d-f9130d346bcb None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:39.782796 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2e3216-db89-4037-847e-cdd53bd31e81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:39.784239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2e3216-db89-4037-847e-cdd53bd31e81 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:39.784706 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2e3216-db89-4037-847e-cdd53bd31e81 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:39.784975 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2e3216-db89-4037-847e-cdd53bd31e81 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:39.795430 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2e3216-db89-4037-847e-cdd53bd31e81 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:39.797003 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2e3216-db89-4037-847e-cdd53bd31e81 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:39.813560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-ac2e3216-db89-4037-847e-cdd53bd31e81 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:39.813560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:39.813560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:39.813560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:39.813560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:39.813560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:39.813560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:39.813560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:39.813560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:39.813560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:39.813560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:39.813560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:39.813560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:39.813560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:39.813560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:39.814562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:39.814562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:39.814562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:39.814562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:39.814562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:39.814562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:39.814562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:39.814562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:39.814562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:39.814562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:39.814562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:39.814562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:39.814562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:39.814562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:39.814562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:39.814562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:39.814562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:39.815550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:39.815550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:39.815550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:39.815550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:39.815550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:39.815550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:39.815550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:39.815550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:39.816008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 342/679] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:39.816961 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2e3216-db89-4037-847e-cdd53bd31e81 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:39.851064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:39.855359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:39.875667 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab43553-5024-48cc-993d-f9130d346bcb None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:39.920553 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61d6160-77cf-47e2-9dff-063ae94f53ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:39.921168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61d6160-77cf-47e2-9dff-063ae94f53ce None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:39.921447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61d6160-77cf-47e2-9dff-063ae94f53ce None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:39.921717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61d6160-77cf-47e2-9dff-063ae94f53ce None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:39.922481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61d6160-77cf-47e2-9dff-063ae94f53ce None admin] RBAC: Adding query filter params (role.id=144dd855f74945548ea65db37acda1c1, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=4e2742f567444dff88ed919be52217de) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:39.922833 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61d6160-77cf-47e2-9dff-063ae94f53ce None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:39.924256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61d6160-77cf-47e2-9dff-063ae94f53ce None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:39.930386 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 338/680] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:27:39 2022] GET /identity/v3/role_assignments?user.id=4e2742f567444dff88ed919be52217de&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=144dd855f74945548ea65db37acda1c1 => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:39.931294 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61d6160-77cf-47e2-9dff-063ae94f53ce None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:39.953298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2e3216-db89-4037-847e-cdd53bd31e81 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:40.024167 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a20253b-c66b-4206-820d-8eacb2c73493 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:40.024731 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a20253b-c66b-4206-820d-8eacb2c73493 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:40.025009 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a20253b-c66b-4206-820d-8eacb2c73493 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:40.025275 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a20253b-c66b-4206-820d-8eacb2c73493 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:40.040294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a20253b-c66b-4206-820d-8eacb2c73493 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:40.041795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a20253b-c66b-4206-820d-8eacb2c73493 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:40.060337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-4a20253b-c66b-4206-820d-8eacb2c73493 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 30 09:27:40.060337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:40.060337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:40.060337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:40.060337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:40.060337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:40.060337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:40.060337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:40.060337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:40.060337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:40.060337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:40.060337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:40.060337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:40.060337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:40.060337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:40.060337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:40.061482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:40.061482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:40.061482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:40.061482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:40.061482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:40.061482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:40.061482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:40.061482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:40.061482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:40.061482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:40.061482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:40.061482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:40.061482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:40.061482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:40.061482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:40.061482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:40.061482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:40.061482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:40.062473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:40.062473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:40.062473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:40.062473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:40.062473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:40.062473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:40.062473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:40.062473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:40.062473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 30 09:27:40.062473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:40.064694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 343/681] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:27:39 2022] GET /identity/v3/users/swift => generated 83 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:40.070806 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a20253b-c66b-4206-820d-8eacb2c73493 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:40.115567 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61d6160-77cf-47e2-9dff-063ae94f53ce None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:40.158472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368db9fb-f07e-41e1-b96b-a850ce91a3ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:40.159240 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368db9fb-f07e-41e1-b96b-a850ce91a3ab None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:40.159517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368db9fb-f07e-41e1-b96b-a850ce91a3ab None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:40.159786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368db9fb-f07e-41e1-b96b-a850ce91a3ab None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:40.160498 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368db9fb-f07e-41e1-b96b-a850ce91a3ab None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:40.160802 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368db9fb-f07e-41e1-b96b-a850ce91a3ab None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:40.162293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368db9fb-f07e-41e1-b96b-a850ce91a3ab None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:40.186693 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 339/682] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:40.189465 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368db9fb-f07e-41e1-b96b-a850ce91a3ab None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:40.195928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:40.243389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:40.250254 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a20253b-c66b-4206-820d-8eacb2c73493 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:40.292031 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cdec094-ac2f-4dd6-9731-4ea1929a2567 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:40.292562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdec094-ac2f-4dd6-9731-4ea1929a2567 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:40.293106 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdec094-ac2f-4dd6-9731-4ea1929a2567 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:40.293454 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdec094-ac2f-4dd6-9731-4ea1929a2567 None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:40.294879 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cdec094-ac2f-4dd6-9731-4ea1929a2567 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:40.295189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cdec094-ac2f-4dd6-9731-4ea1929a2567 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:40.296963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cdec094-ac2f-4dd6-9731-4ea1929a2567 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:40.312383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368db9fb-f07e-41e1-b96b-a850ce91a3ab None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:40.319564 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 344/683] 10.222.0.39 () {56 vars in 1135 bytes} [Fri Sep 30 09:27:40 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:40.326114 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cdec094-ac2f-4dd6-9731-4ea1929a2567 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:40.355520 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb372c5-d88b-47f9-a475-49aa2858e669 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:40.356474 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb372c5-d88b-47f9-a475-49aa2858e669 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:40.356758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb372c5-d88b-47f9-a475-49aa2858e669 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:40.357091 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb372c5-d88b-47f9-a475-49aa2858e669 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:40.357814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb372c5-d88b-47f9-a475-49aa2858e669 None admin] RBAC: Adding query filter params (role.id=144dd855f74945548ea65db37acda1c1, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=9266babd21414ae9998716e5a0478e9d) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:40.358384 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb372c5-d88b-47f9-a475-49aa2858e669 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:40.359882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb372c5-d88b-47f9-a475-49aa2858e669 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:40.370689 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 340/684] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:27:40 2022] GET /identity/v3/role_assignments?user.id=9266babd21414ae9998716e5a0478e9d&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=144dd855f74945548ea65db37acda1c1 => generated 608 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:40.440440 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cdec094-ac2f-4dd6-9731-4ea1929a2567 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:40.480140 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e6e729-3f0d-447f-866b-e92027b3ed0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:40.480852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e6e729-3f0d-447f-866b-e92027b3ed0a None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:40.481167 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e6e729-3f0d-447f-866b-e92027b3ed0a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:40.481509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e6e729-3f0d-447f-866b-e92027b3ed0a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:40.508795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:40.564915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e6e729-3f0d-447f-866b-e92027b3ed0a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:40.566305 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e6e729-3f0d-447f-866b-e92027b3ed0a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:40.669756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 345/685] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:27:40.675313 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb372c5-d88b-47f9-a475-49aa2858e669 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:40.750198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:40.788642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb372c5-d88b-47f9-a475-49aa2858e669 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:40.826931 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36d70e1-41db-4ea0-9086-5f4c88b36a45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:40.827492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36d70e1-41db-4ea0-9086-5f4c88b36a45 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:40.827772 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36d70e1-41db-4ea0-9086-5f4c88b36a45 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:40.828042 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36d70e1-41db-4ea0-9086-5f4c88b36a45 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:40.839236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36d70e1-41db-4ea0-9086-5f4c88b36a45 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:40.840712 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36d70e1-41db-4ea0-9086-5f4c88b36a45 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:40.854985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-d36d70e1-41db-4ea0-9086-5f4c88b36a45 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:40.854985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:40.854985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:40.854985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:40.854985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:40.854985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:40.854985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:40.854985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:40.854985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:40.854985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:40.854985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:40.854985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:40.854985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:40.854985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:40.854985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:40.854985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:40.856014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:40.856014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:40.856014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:40.856014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:40.856014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:40.856014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:40.856014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:40.856014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:40.856014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:40.856014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:40.856014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:40.856014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:40.856014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:40.856014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:40.856014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:40.856014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:40.856014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:40.856919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:40.856919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:40.856919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:40.857836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 341/686] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:40.863142 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e6e729-3f0d-447f-866b-e92027b3ed0a None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:40.948596 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:40.999855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e6e729-3f0d-447f-866b-e92027b3ed0a None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:41.046106 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3e7227-c765-41e9-9f92-65beda12e252 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:41.046745 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3e7227-c765-41e9-9f92-65beda12e252 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:41.047033 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3e7227-c765-41e9-9f92-65beda12e252 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:41.047302 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3e7227-c765-41e9-9f92-65beda12e252 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:41.047938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3e7227-c765-41e9-9f92-65beda12e252 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:41.048240 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3e7227-c765-41e9-9f92-65beda12e252 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:41.049610 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3e7227-c765-41e9-9f92-65beda12e252 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:41.066282 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 346/687] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:41.071738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36d70e1-41db-4ea0-9086-5f4c88b36a45 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:41.176829 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36d70e1-41db-4ea0-9086-5f4c88b36a45 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:41.199279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:41.222144 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1ca5d4-4795-4edb-a2d7-676d42034935 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:41.222832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1ca5d4-4795-4edb-a2d7-676d42034935 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:41.223115 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1ca5d4-4795-4edb-a2d7-676d42034935 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:41.223391 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1ca5d4-4795-4edb-a2d7-676d42034935 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:41.232140 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1ca5d4-4795-4edb-a2d7-676d42034935 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:41.233698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1ca5d4-4795-4edb-a2d7-676d42034935 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:41.251757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-2f1ca5d4-4795-4edb-a2d7-676d42034935 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:41.251757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:41.251757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:41.251757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:41.251757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:41.251757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:41.251757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:41.251757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:41.251757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:41.251757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:41.251757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:41.251757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:41.251757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:41.251757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:41.251757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:41.252923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:41.252923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:41.252923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:41.252923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:41.252923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:41.252923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:41.252923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:41.252923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:41.252923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:41.252923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:41.252923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:41.252923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:41.252923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:41.252923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:41.252923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:41.252923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:41.252923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:41.254036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:41.254036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:41.254036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:41.254036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:41.254036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:41.254036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:41.254036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:41.254036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:41.258194 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 342/688] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:41.264213 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3e7227-c765-41e9-9f92-65beda12e252 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:41.368171 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3e7227-c765-41e9-9f92-65beda12e252 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:41.383499 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:41.404471 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e01bb7fe-540d-41b1-9cb7-6d4ad1c5990c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:41.405059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01bb7fe-540d-41b1-9cb7-6d4ad1c5990c None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:41.405342 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01bb7fe-540d-41b1-9cb7-6d4ad1c5990c None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:41.405615 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01bb7fe-540d-41b1-9cb7-6d4ad1c5990c None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:41.406298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e01bb7fe-540d-41b1-9cb7-6d4ad1c5990c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:41.406624 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e01bb7fe-540d-41b1-9cb7-6d4ad1c5990c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:41.408154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e01bb7fe-540d-41b1-9cb7-6d4ad1c5990c None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:41.428062 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 347/689] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:41.433937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1ca5d4-4795-4edb-a2d7-676d42034935 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:41.563435 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1ca5d4-4795-4edb-a2d7-676d42034935 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:41.598890 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0252f9db-d75e-4eda-8806-fac61cb65f58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:41.599498 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0252f9db-d75e-4eda-8806-fac61cb65f58 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:41.599776 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0252f9db-d75e-4eda-8806-fac61cb65f58 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:41.600050 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0252f9db-d75e-4eda-8806-fac61cb65f58 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:41.600766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0252f9db-d75e-4eda-8806-fac61cb65f58 None admin] RBAC: Adding query filter params (role.id=144dd855f74945548ea65db37acda1c1, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=0e5ad921e6fa41c297696ec7279f0fc4) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:41.601071 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0252f9db-d75e-4eda-8806-fac61cb65f58 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:41.602487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0252f9db-d75e-4eda-8806-fac61cb65f58 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:41.609325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 343/690] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:27:41 2022] GET /identity/v3/role_assignments?user.id=0e5ad921e6fa41c297696ec7279f0fc4&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=144dd855f74945548ea65db37acda1c1 => generated 608 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:41.675405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23e33a1-11bf-4d6e-adcf-8ed75e3bfedf None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:41.675723 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23e33a1-11bf-4d6e-adcf-8ed75e3bfedf None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:41.675962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23e33a1-11bf-4d6e-adcf-8ed75e3bfedf None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:41.677449 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 348/691] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:41.683274 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4870af10-22b4-4609-b15c-0bfd251852ff None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:41.683554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4870af10-22b4-4609-b15c-0bfd251852ff None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:41.683745 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4870af10-22b4-4609-b15c-0bfd251852ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:41.708241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-4870af10-22b4-4609-b15c-0bfd251852ff None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:41.709480 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4870af10-22b4-4609-b15c-0bfd251852ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:41.759263 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:41.801743 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 344/692] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:42.117860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23e33a1-11bf-4d6e-adcf-8ed75e3bfedf None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:42.237968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23e33a1-11bf-4d6e-adcf-8ed75e3bfedf None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:42.273957 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee5b957-3f2d-4396-b38a-ab8ca50a297f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:42.274612 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee5b957-3f2d-4396-b38a-ab8ca50a297f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:42.274972 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee5b957-3f2d-4396-b38a-ab8ca50a297f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:42.275249 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee5b957-3f2d-4396-b38a-ab8ca50a297f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:42.286961 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee5b957-3f2d-4396-b38a-ab8ca50a297f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:42.288436 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee5b957-3f2d-4396-b38a-ab8ca50a297f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:42.303963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-dee5b957-3f2d-4396-b38a-ab8ca50a297f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:42.303963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:42.303963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:42.303963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:42.303963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:42.303963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:42.303963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:42.303963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:42.303963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:42.303963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:42.303963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:42.303963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:42.303963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:42.303963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:42.303963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:42.303963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:42.304934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:42.304934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:42.304934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:42.304934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:42.304934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:42.304934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:42.304934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:42.304934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:42.304934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:42.304934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:42.304934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:42.304934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:42.304934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:42.304934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:42.304934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:42.304934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:42.304934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:42.305846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:42.305846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:42.305846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:42.305846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 349/693] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:42.312993 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4870af10-22b4-4609-b15c-0bfd251852ff None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:42.440989 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4870af10-22b4-4609-b15c-0bfd251852ff None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:42.479989 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ce0110-9c6e-418e-ba5e-40359c132679 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:42.480633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ce0110-9c6e-418e-ba5e-40359c132679 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:42.481169 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ce0110-9c6e-418e-ba5e-40359c132679 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:42.482932 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 350/694] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:27:42.487364 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4edf6cd-b89d-497a-864a-bade58f26b22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:42.487985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4edf6cd-b89d-497a-864a-bade58f26b22 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:42.488267 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4edf6cd-b89d-497a-864a-bade58f26b22 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:42.488541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4edf6cd-b89d-497a-864a-bade58f26b22 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:42.488888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b4f196-9c47-4289-9e97-98765f5b1143 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:42.489185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4edf6cd-b89d-497a-864a-bade58f26b22 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:42.489488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4edf6cd-b89d-497a-864a-bade58f26b22 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:42.489774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b4f196-9c47-4289-9e97-98765f5b1143 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:42.490234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b4f196-9c47-4289-9e97-98765f5b1143 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:42.490889 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4edf6cd-b89d-497a-864a-bade58f26b22 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:42.510352 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 345/695] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:42.511710 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4edf6cd-b89d-497a-864a-bade58f26b22 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:42.535736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-f1b4f196-9c47-4289-9e97-98765f5b1143 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:42.537374 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-f1b4f196-9c47-4289-9e97-98765f5b1143 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:42.543741 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:42.573984 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:42.617854 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4edf6cd-b89d-497a-864a-bade58f26b22 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:42.669770 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8578701-0381-4abb-a758-0f2cba4560bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:42.670411 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8578701-0381-4abb-a758-0f2cba4560bb None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:42.670714 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8578701-0381-4abb-a758-0f2cba4560bb None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:42.671037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8578701-0381-4abb-a758-0f2cba4560bb None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:42.671728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8578701-0381-4abb-a758-0f2cba4560bb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:42.675539 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8578701-0381-4abb-a758-0f2cba4560bb None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:42.685860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 351/696] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:27:42.731371 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7228f6-a8d9-438f-bcd1-9cd04039d9d7 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:42.731671 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7228f6-a8d9-438f-bcd1-9cd04039d9d7 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:42.731933 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7228f6-a8d9-438f-bcd1-9cd04039d9d7 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:42.733646 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 352/697] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:27:42.738503 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7aa9b65-f59f-4e44-8364-4e4a1b511637 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:42.738809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7aa9b65-f59f-4e44-8364-4e4a1b511637 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:42.739086 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7aa9b65-f59f-4e44-8364-4e4a1b511637 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:42.761612 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-c7aa9b65-f59f-4e44-8364-4e4a1b511637 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:42.762912 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c7aa9b65-f59f-4e44-8364-4e4a1b511637 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:42.805705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 346/698] 10.222.0.39 () {60 vars in 1132 bytes} [Fri Sep 30 09:27:42 2022] POST /identity/v3/users => generated 254 bytes in 294 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:27:42.837387 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 353/699] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 30 09:27:42.974713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8578701-0381-4abb-a758-0f2cba4560bb None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:43.071097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:43.084562 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8578701-0381-4abb-a758-0f2cba4560bb None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:43.123515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f64c562-d83a-49c7-9e38-12bbe254f421 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:43.124524 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f64c562-d83a-49c7-9e38-12bbe254f421 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:43.125006 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f64c562-d83a-49c7-9e38-12bbe254f421 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:43.125497 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f64c562-d83a-49c7-9e38-12bbe254f421 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:43.139688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f64c562-d83a-49c7-9e38-12bbe254f421 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:43.145239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-0f64c562-d83a-49c7-9e38-12bbe254f421 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:27:43.145239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:43.145239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:43.145239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:43.145239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:43.145239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:43.145239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:43.145239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:43.145239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:43.145239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:43.145239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:43.145239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:43.145239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:43.145239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:43.145239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:43.145239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:43.146192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:43.146192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:43.146192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:43.146192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:43.146192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:43.146192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:43.146192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:43.146192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:43.146192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:43.146192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:43.146192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:43.146192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:43.146192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:43.146192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:43.146192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:43.146192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:43.147081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:43.147081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:43.147081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:43.147081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:43.147081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:43.147081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:43.147081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:43.147081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:43.147081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:43.147081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:43.147081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:27:43.147081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:43.149419 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 347/700] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:27:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:43.157195 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7aa9b65-f59f-4e44-8364-4e4a1b511637 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:43.198852 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f64c562-d83a-49c7-9e38-12bbe254f421 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:43.276065 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:43.299274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7aa9b65-f59f-4e44-8364-4e4a1b511637 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:43.331665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f64c562-d83a-49c7-9e38-12bbe254f421 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:43.350073 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69413779-4321-418b-8e5f-0b5e99ba7ba5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:43.351208 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69413779-4321-418b-8e5f-0b5e99ba7ba5 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:43.351640 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69413779-4321-418b-8e5f-0b5e99ba7ba5 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:43.351915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69413779-4321-418b-8e5f-0b5e99ba7ba5 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:43.352871 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69413779-4321-418b-8e5f-0b5e99ba7ba5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:43.353174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69413779-4321-418b-8e5f-0b5e99ba7ba5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:43.354552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69413779-4321-418b-8e5f-0b5e99ba7ba5 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:43.366091 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 354/701] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:27:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:43.371324 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69413779-4321-418b-8e5f-0b5e99ba7ba5 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:43.372378 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d359b872-1924-47a1-a8be-7de98213c973 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:43.373015 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d359b872-1924-47a1-a8be-7de98213c973 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:43.373293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d359b872-1924-47a1-a8be-7de98213c973 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:43.373566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d359b872-1924-47a1-a8be-7de98213c973 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:43.374144 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d359b872-1924-47a1-a8be-7de98213c973 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:43.375539 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d359b872-1924-47a1-a8be-7de98213c973 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:43.386664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-d359b872-1924-47a1-a8be-7de98213c973 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 30 09:27:43.386664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:43.386664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:27:43.386664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:27:43.386664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:27:43.386664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:27:43.386664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:27:43.386664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:27:43.386664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 30 09:27:43.386664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:43.386664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:27:43.386664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:43.386664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:43.386664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:43.386664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:43.386664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:43.386664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:43.388521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:43.388521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:43.388521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:43.388521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:43.388521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:43.388521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:43.388521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:27:43.388521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:27:43.388521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:27:43.388521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:27:43.388521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:43.388521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:43.388521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:43.388521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:43.388521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:43.388521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:43.389453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:43.389453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:43.389453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:43.389453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:43.389453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:43.389453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:43.389453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:43.389453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:43.389453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:43.389453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:43.389453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:27:43.389453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:27:43.389453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 30 09:27:43.389453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:43.393068 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 348/702] 10.222.0.39 () {56 vars in 1129 bytes} [Fri Sep 30 09:27:43 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:43.394193 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d359b872-1924-47a1-a8be-7de98213c973 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:43.411545 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:43.499180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69413779-4321-418b-8e5f-0b5e99ba7ba5 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:43.514930 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d359b872-1924-47a1-a8be-7de98213c973 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:43.535246 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bcb0c1-6d21-480b-90b1-da5c4da71cee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:43.536008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bcb0c1-6d21-480b-90b1-da5c4da71cee None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:43.536477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bcb0c1-6d21-480b-90b1-da5c4da71cee None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:43.536744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bcb0c1-6d21-480b-90b1-da5c4da71cee None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:43.547868 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4bcb0c1-6d21-480b-90b1-da5c4da71cee None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:43.549428 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4bcb0c1-6d21-480b-90b1-da5c4da71cee None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:43.557496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e843e1-c768-4531-b371-66fa60284f21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:43.558405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:43.559421 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e843e1-c768-4531-b371-66fa60284f21 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:43.559943 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e843e1-c768-4531-b371-66fa60284f21 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:43.560364 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e843e1-c768-4531-b371-66fa60284f21 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:43.561144 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e843e1-c768-4531-b371-66fa60284f21 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:43.561594 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e843e1-c768-4531-b371-66fa60284f21 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:43.563138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e843e1-c768-4531-b371-66fa60284f21 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:43.565659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-c4bcb0c1-6d21-480b-90b1-da5c4da71cee None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:43.565659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:43.565659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:43.565659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:43.565659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:43.565659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:43.565659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:43.565659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:43.565659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:43.565659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:43.565659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:43.565659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:43.565659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:43.565659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:43.565659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:43.565659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:43.571052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:43.571052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:43.571052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:43.571052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:43.571052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:43.571052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:43.571052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:43.571052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:43.571052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:43.571052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:43.571052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:43.571052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:43.571052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:43.571052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:43.571052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:43.571052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:43.571052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:43.572006 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:43.572006 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:43.572006 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:43.572182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 349/704] 10.222.0.39 () {56 vars in 1124 bytes} [Fri Sep 30 09:27:43 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:43.572182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d729a767-c740-483e-9953-c0b1c141c264 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:43.572409 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 355/704] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:43.572757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d729a767-c740-483e-9953-c0b1c141c264 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:43.573183 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d729a767-c740-483e-9953-c0b1c141c264 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:43.573542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bcb0c1-6d21-480b-90b1-da5c4da71cee None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:43.575622 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 350/705] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:43 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:43.580411 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d729a767-c740-483e-9953-c0b1c141c264 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:43.685862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bcb0c1-6d21-480b-90b1-da5c4da71cee None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:43.726245 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e9caab-e253-48f5-baaa-8e3acf22fbf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:43.727059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e9caab-e253-48f5-baaa-8e3acf22fbf1 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:43.727433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e9caab-e253-48f5-baaa-8e3acf22fbf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:43.727800 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e9caab-e253-48f5-baaa-8e3acf22fbf1 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:43.728555 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e9caab-e253-48f5-baaa-8e3acf22fbf1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:43.728956 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e9caab-e253-48f5-baaa-8e3acf22fbf1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:43.730494 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e9caab-e253-48f5-baaa-8e3acf22fbf1 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:43.749792 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 356/706] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:43.751484 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c152d094-1018-4426-a345-7bcd5005c821 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:43.751869 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c152d094-1018-4426-a345-7bcd5005c821 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:43.752237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c152d094-1018-4426-a345-7bcd5005c821 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:43.773002 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-c152d094-1018-4426-a345-7bcd5005c821 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:43.774371 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c152d094-1018-4426-a345-7bcd5005c821 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:43.792969 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:43.818837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d729a767-c740-483e-9953-c0b1c141c264 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:43.835928 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:43.847942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 357/707] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:43.849132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c152d094-1018-4426-a345-7bcd5005c821 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:43.872667 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee27cb7-fce3-44c9-8b9e-7a6e651839ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:43.872992 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee27cb7-fce3-44c9-8b9e-7a6e651839ff None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:43.873258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee27cb7-fce3-44c9-8b9e-7a6e651839ff None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:43.873528 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee27cb7-fce3-44c9-8b9e-7a6e651839ff None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:43.874155 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee27cb7-fce3-44c9-8b9e-7a6e651839ff None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:43.874454 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee27cb7-fce3-44c9-8b9e-7a6e651839ff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:43.875846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee27cb7-fce3-44c9-8b9e-7a6e651839ff None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:43.891034 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 351/708] 10.222.0.39 () {56 vars in 1124 bytes} [Fri Sep 30 09:27:43 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:43.984762 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c152d094-1018-4426-a345-7bcd5005c821 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:44.027564 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dc0b18-50ed-4733-b0b0-0635c835506b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:44.028539 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dc0b18-50ed-4733-b0b0-0635c835506b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:44.029114 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dc0b18-50ed-4733-b0b0-0635c835506b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:44.029607 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dc0b18-50ed-4733-b0b0-0635c835506b None admin] PATH_INFO: `/v3/users/nova` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:44.037675 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57dc0b18-50ed-4733-b0b0-0635c835506b None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:44.039645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57dc0b18-50ed-4733-b0b0-0635c835506b None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:44.053015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-57dc0b18-50ed-4733-b0b0-0635c835506b None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 30 09:27:44.053015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:44.053015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:44.053015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:44.053015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:44.053015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:44.053015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:44.053015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:44.053015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:44.053015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:44.053015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:44.053015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:44.053015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:44.053015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:44.053015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:44.053015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:44.054050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:44.054050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:44.054050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:44.054050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:44.054050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:44.054050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:44.054050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:44.054050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:44.054050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:44.054050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:44.054050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:44.054050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:44.054050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:44.054050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:44.054050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:44.054050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:44.054050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:44.054050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:44.055039 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:44.055039 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:44.055039 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:44.055039 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:44.055039 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:44.055039 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:44.055039 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:44.055039 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:44.055039 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 30 09:27:44.055039 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:44.057306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 358/709] 10.222.0.39 () {56 vars in 1093 bytes} [Fri Sep 30 09:27:43 2022] GET /identity/v3/users/nova => generated 82 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:44.065808 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee27cb7-fce3-44c9-8b9e-7a6e651839ff None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:44.204653 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee27cb7-fce3-44c9-8b9e-7a6e651839ff None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:44.231623 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dc0b18-50ed-4733-b0b0-0635c835506b None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:44.241318 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d75ef20-724a-4aa0-a3b0-90f58411960c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:44.241865 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d75ef20-724a-4aa0-a3b0-90f58411960c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:44.242143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d75ef20-724a-4aa0-a3b0-90f58411960c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:44.242414 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d75ef20-724a-4aa0-a3b0-90f58411960c None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:44.243178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d75ef20-724a-4aa0-a3b0-90f58411960c None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:44.243483 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d75ef20-724a-4aa0-a3b0-90f58411960c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:44.244906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d75ef20-724a-4aa0-a3b0-90f58411960c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:44.255348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:44.264742 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 352/710] 10.222.0.39 () {56 vars in 1133 bytes} [Fri Sep 30 09:27:44 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:44.270116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d75ef20-724a-4aa0-a3b0-90f58411960c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:44.312915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:44.408951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dc0b18-50ed-4733-b0b0-0635c835506b None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:44.460846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d75ef20-724a-4aa0-a3b0-90f58411960c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:44.471285 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a037c5-3e57-48ff-9f7a-d91fa43c0f41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:44.471927 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a037c5-3e57-48ff-9f7a-d91fa43c0f41 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:44.472211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a037c5-3e57-48ff-9f7a-d91fa43c0f41 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:44.472480 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a037c5-3e57-48ff-9f7a-d91fa43c0f41 None admin] PATH_INFO: `/v3/services/network` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:44.473192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a037c5-3e57-48ff-9f7a-d91fa43c0f41 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:44.474436 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a037c5-3e57-48ff-9f7a-d91fa43c0f41 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:44.498852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-24a037c5-3e57-48ff-9f7a-d91fa43c0f41 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 30 09:27:44.498852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:44.498852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:27:44.498852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:27:44.498852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:27:44.498852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:27:44.498852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:27:44.498852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:27:44.498852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 30 09:27:44.498852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:44.498852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:27:44.498852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:44.498852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:44.498852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:44.498852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:44.498852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:44.498852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:44.499953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:44.499953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:44.499953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:44.499953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:44.499953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:44.499953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:44.499953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:27:44.499953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:27:44.499953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:27:44.499953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:27:44.499953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:44.499953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:44.499953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:44.499953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:44.499953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:44.499953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:44.500829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:44.500829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:44.500829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:44.500829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:44.500829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:44.500829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:44.500829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:44.500829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:44.500829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:44.500829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:44.500829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:27:44.500829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:27:44.500829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 30 09:27:44.500829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:44.500829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 359/711] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:44 2022] GET /identity/v3/services/network => generated 88 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:44.508506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a037c5-3e57-48ff-9f7a-d91fa43c0f41 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:44.512195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17effecd-633c-4faa-bff6-cc8158cedaf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:44.513093 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17effecd-633c-4faa-bff6-cc8158cedaf1 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:44.513579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17effecd-633c-4faa-bff6-cc8158cedaf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:44.514055 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17effecd-633c-4faa-bff6-cc8158cedaf1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:44.583915 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:44.588825 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17effecd-633c-4faa-bff6-cc8158cedaf1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:44.591132 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17effecd-633c-4faa-bff6-cc8158cedaf1 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:44.702350 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 353/712] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:27:44.707515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17effecd-633c-4faa-bff6-cc8158cedaf1 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:44.736717 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a037c5-3e57-48ff-9f7a-d91fa43c0f41 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:44.769917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:44.803165 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5a4a74-4d56-4ce5-9ded-8ad074ccf992 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:44.803781 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5a4a74-4d56-4ce5-9ded-8ad074ccf992 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:44.804064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5a4a74-4d56-4ce5-9ded-8ad074ccf992 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:44.804337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5a4a74-4d56-4ce5-9ded-8ad074ccf992 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:44.804964 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5a4a74-4d56-4ce5-9ded-8ad074ccf992 None admin] RBAC: Adding query filter params (name=network) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:44.805268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5a4a74-4d56-4ce5-9ded-8ad074ccf992 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:44.806663 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5a4a74-4d56-4ce5-9ded-8ad074ccf992 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:44.813765 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 360/713] 10.222.0.39 () {56 vars in 1112 bytes} [Fri Sep 30 09:27:44 2022] GET /identity/v3/services?name=network => generated 124 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:44.818552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8545307f-56ae-4a9e-85a7-b67c72fa905f None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:44.818552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8545307f-56ae-4a9e-85a7-b67c72fa905f None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:44.818552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8545307f-56ae-4a9e-85a7-b67c72fa905f None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:44.818552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 361/714] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:44.819613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8545307f-56ae-4a9e-85a7-b67c72fa905f None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:44.825147 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:44.829812 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17effecd-633c-4faa-bff6-cc8158cedaf1 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:44.870669 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b89bcc1-44f9-4c20-9b79-ebfc704c743c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:44.872292 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b89bcc1-44f9-4c20-9b79-ebfc704c743c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:44.872796 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b89bcc1-44f9-4c20-9b79-ebfc704c743c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:44.873366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b89bcc1-44f9-4c20-9b79-ebfc704c743c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:44.885756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b89bcc1-44f9-4c20-9b79-ebfc704c743c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:44.887659 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b89bcc1-44f9-4c20-9b79-ebfc704c743c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:44.905515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-6b89bcc1-44f9-4c20-9b79-ebfc704c743c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:44.905515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:44.905515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:44.905515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:44.905515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:44.905515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:44.905515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:44.905515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:44.905515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:44.905515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:44.905515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:44.905515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:44.905515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:44.905515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:44.905515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:44.905515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:44.906558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:44.906558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:44.906558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:44.906558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:44.906558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:44.906558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:44.906558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:44.906558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:44.906558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:44.906558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:44.906558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:44.906558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:44.906558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:44.906558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:44.906558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:44.906558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:44.906558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:44.907559 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:44.907559 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:44.907559 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:44.909001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 354/715] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:44.909301 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d250c3d2-9027-415e-9185-e44b32259b59 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:44.909582 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d250c3d2-9027-415e-9185-e44b32259b59 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:44.909853 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d250c3d2-9027-415e-9185-e44b32259b59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:44.929156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d250c3d2-9027-415e-9185-e44b32259b59 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:44.930348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d250c3d2-9027-415e-9185-e44b32259b59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:44.948265 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8545307f-56ae-4a9e-85a7-b67c72fa905f None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:44.987162 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9438a7-8d6a-4d7b-86ca-5f55bf408a31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:44.988205 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9438a7-8d6a-4d7b-86ca-5f55bf408a31 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:44.988712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9438a7-8d6a-4d7b-86ca-5f55bf408a31 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:44.989190 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9438a7-8d6a-4d7b-86ca-5f55bf408a31 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:44.990093 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9438a7-8d6a-4d7b-86ca-5f55bf408a31 None admin] RBAC: Adding query filter params (type=network) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:44.990718 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9438a7-8d6a-4d7b-86ca-5f55bf408a31 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:44.992460 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9438a7-8d6a-4d7b-86ca-5f55bf408a31 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:45.004355 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 362/716] 10.222.0.39 () {56 vars in 1112 bytes} [Fri Sep 30 09:27:44 2022] GET /identity/v3/services?type=network => generated 124 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:45.004355 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9438a7-8d6a-4d7b-86ca-5f55bf408a31 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:45.009315 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 355/717] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:45.069641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:45.255346 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9438a7-8d6a-4d7b-86ca-5f55bf408a31 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:45.304673 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e9a451-be0d-479f-97a6-be3ee6f1c3c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:45.305284 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e9a451-be0d-479f-97a6-be3ee6f1c3c9 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:45.305567 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e9a451-be0d-479f-97a6-be3ee6f1c3c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:45.305841 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e9a451-be0d-479f-97a6-be3ee6f1c3c9 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:45.306482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e9a451-be0d-479f-97a6-be3ee6f1c3c9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:45.306809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e9a451-be0d-479f-97a6-be3ee6f1c3c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:45.308191 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e9a451-be0d-479f-97a6-be3ee6f1c3c9 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:45.322556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 363/718] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:45.327602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d250c3d2-9027-415e-9185-e44b32259b59 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:45.366139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e9a451-be0d-479f-97a6-be3ee6f1c3c9 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:45.479027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:45.503625 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d250c3d2-9027-415e-9185-e44b32259b59 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:45.600186 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e9a451-be0d-479f-97a6-be3ee6f1c3c9 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:45.600363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0b8d65-ed13-4a93-9c72-5e4695674dca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:45.600363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0b8d65-ed13-4a93-9c72-5e4695674dca None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:45.600363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0b8d65-ed13-4a93-9c72-5e4695674dca None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:45.600363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0b8d65-ed13-4a93-9c72-5e4695674dca None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:45.600977 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:45.601061 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0b8d65-ed13-4a93-9c72-5e4695674dca None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:45.601061 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0b8d65-ed13-4a93-9c72-5e4695674dca None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:45.614640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-3d0b8d65-ed13-4a93-9c72-5e4695674dca None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:45.614640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:45.614640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:45.614640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:45.614640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:45.614640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:45.614640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:45.614640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:45.614640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:45.614640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:45.614640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:45.614640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:45.614640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:45.614640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:45.614640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:45.615608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:45.615608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:45.615608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:45.615608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:45.615608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:45.615608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:45.615608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:45.615608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:45.615608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:45.615608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:45.615608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:45.615608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:45.615608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:45.615608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:45.615608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:45.615608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:45.615608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:45.616932 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:45.616932 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:45.616932 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:45.616932 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:45.616932 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:45.616932 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:45.616932 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:45.616932 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:45.616932 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 356/719] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:45.617425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c95013-4edb-45c4-9838-341519d63e7e None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:45.617696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c95013-4edb-45c4-9838-341519d63e7e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:45.617897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c95013-4edb-45c4-9838-341519d63e7e None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:45.619572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 357/720] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:45.626427 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c95013-4edb-45c4-9838-341519d63e7e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:45.639821 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1ddd8f-58ab-4376-aa95-0aa465ea4f36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:45.641283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1ddd8f-58ab-4376-aa95-0aa465ea4f36 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:45.641799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1ddd8f-58ab-4376-aa95-0aa465ea4f36 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:45.642269 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1ddd8f-58ab-4376-aa95-0aa465ea4f36 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:45.656829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1ddd8f-58ab-4376-aa95-0aa465ea4f36 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:45.664044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-aa1ddd8f-58ab-4376-aa95-0aa465ea4f36 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:27:45.664044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:45.664044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:45.664044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:45.664044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:45.664044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:45.664044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:45.664044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:45.664044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:45.664044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:45.664044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:45.664044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:45.664044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:45.664044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:45.664044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:45.664044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:45.664994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:45.664994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:45.664994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:45.664994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:45.664994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:45.664994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:45.664994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:45.664994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:45.664994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:45.664994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:45.664994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:45.664994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:45.664994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:45.664994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:45.664994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:45.664994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:45.665855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:45.665855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:45.665855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:45.665855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:45.665855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:45.665855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:45.665855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:45.665855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:45.665855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:45.665855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:45.665855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:27:45.665855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:45.673571 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 364/721] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:27:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:45.675395 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf13f207-ebc0-457f-a129-3d5f4790e6d2 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:45.675685 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf13f207-ebc0-457f-a129-3d5f4790e6d2 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:45.675989 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf13f207-ebc0-457f-a129-3d5f4790e6d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:45.697433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-bf13f207-ebc0-457f-a129-3d5f4790e6d2 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:45.699124 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-bf13f207-ebc0-457f-a129-3d5f4790e6d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:45.752707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c95013-4edb-45c4-9838-341519d63e7e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:45.781812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 365/722] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:45.783084 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf13f207-ebc0-457f-a129-3d5f4790e6d2 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:45.804019 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7413504e-f256-4f7c-9e2c-779f74023350 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:45.805624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7413504e-f256-4f7c-9e2c-779f74023350 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:45.806455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7413504e-f256-4f7c-9e2c-779f74023350 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:45.807741 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7413504e-f256-4f7c-9e2c-779f74023350 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:45.809991 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7413504e-f256-4f7c-9e2c-779f74023350 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:45.810540 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7413504e-f256-4f7c-9e2c-779f74023350 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:45.813076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7413504e-f256-4f7c-9e2c-779f74023350 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:45.831361 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:45.839167 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 358/723] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:45.850162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7413504e-f256-4f7c-9e2c-779f74023350 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:45.919134 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:45.936459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf13f207-ebc0-457f-a129-3d5f4790e6d2 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:45.969759 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7413504e-f256-4f7c-9e2c-779f74023350 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:46.004083 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bca4fd-82d7-4846-8acc-244b8fa420c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:46.004970 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bca4fd-82d7-4846-8acc-244b8fa420c0 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:46.005479 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bca4fd-82d7-4846-8acc-244b8fa420c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:46.005924 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bca4fd-82d7-4846-8acc-244b8fa420c0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:46.007174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bca4fd-82d7-4846-8acc-244b8fa420c0 None admin] RBAC: Adding query filter params (role.id=e874c311084c4c9fb7691bce461d319b, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=7ff68fb87700423ab53c90520a6bd3b9) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:46.007660 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bca4fd-82d7-4846-8acc-244b8fa420c0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:46.009310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bca4fd-82d7-4846-8acc-244b8fa420c0 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:46.015987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 359/724] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:27:45 2022] GET /identity/v3/role_assignments?user.id=7ff68fb87700423ab53c90520a6bd3b9&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=e874c311084c4c9fb7691bce461d319b => generated 259 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:46.051499 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d0915e-6833-4b4e-891f-8c50a0ff8aa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:46.052144 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d0915e-6833-4b4e-891f-8c50a0ff8aa9 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:46.052423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d0915e-6833-4b4e-891f-8c50a0ff8aa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:46.052691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d0915e-6833-4b4e-891f-8c50a0ff8aa9 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:46.053374 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d0915e-6833-4b4e-891f-8c50a0ff8aa9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:46.053676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d0915e-6833-4b4e-891f-8c50a0ff8aa9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:46.055155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d0915e-6833-4b4e-891f-8c50a0ff8aa9 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:46.070803 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 366/725] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:27:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:46.077010 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bca4fd-82d7-4846-8acc-244b8fa420c0 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:46.137447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d0915e-6833-4b4e-891f-8c50a0ff8aa9 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:46.169985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:46.210821 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bca4fd-82d7-4846-8acc-244b8fa420c0 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:46.253201 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3175f2d0-337f-4cf3-8b75-77caf2d86b55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:46.253839 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3175f2d0-337f-4cf3-8b75-77caf2d86b55 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:46.254115 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3175f2d0-337f-4cf3-8b75-77caf2d86b55 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:46.254388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3175f2d0-337f-4cf3-8b75-77caf2d86b55 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:46.270417 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3175f2d0-337f-4cf3-8b75-77caf2d86b55 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:46.271882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3175f2d0-337f-4cf3-8b75-77caf2d86b55 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:46.286918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-3175f2d0-337f-4cf3-8b75-77caf2d86b55 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:46.286918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:46.286918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:46.286918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:46.286918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:46.286918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:46.286918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:46.286918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:46.286918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:46.286918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:46.286918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:46.286918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:46.286918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:46.286918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:46.286918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:46.286918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:46.287897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:46.287897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:46.287897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:46.287897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:46.287897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:46.287897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:46.287897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:46.287897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:46.287897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:46.287897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:46.287897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:46.287897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:46.287897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:46.287897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:46.287897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:46.287897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:46.287897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:46.289057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:46.289057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:46.289057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:46.289057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 360/726] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:46.294928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3175f2d0-337f-4cf3-8b75-77caf2d86b55 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:46.340909 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d0915e-6833-4b4e-891f-8c50a0ff8aa9 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:46.410551 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfb05e5-fd40-4da1-912f-d0f097d19e13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:46.411219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfb05e5-fd40-4da1-912f-d0f097d19e13 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:46.411499 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfb05e5-fd40-4da1-912f-d0f097d19e13 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:46.411771 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfb05e5-fd40-4da1-912f-d0f097d19e13 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:46.427223 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfb05e5-fd40-4da1-912f-d0f097d19e13 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:46.434726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-1bfb05e5-fd40-4da1-912f-d0f097d19e13 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:46.434726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:46.434726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:46.434726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:46.434726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:46.434726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:46.434726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:46.434726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:46.434726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:46.434726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:46.434726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:46.434726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:46.434726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:46.434726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:46.434726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:46.434726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:46.436638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:46.436638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:46.436638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:46.436638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:46.436638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:46.436638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:46.436638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:46.436638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:46.436638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:46.436638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:46.436638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:46.436638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:46.436638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:46.436638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:46.436638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:46.436638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:46.437688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:46.437688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:46.437688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:46.437688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:46.437688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:46.437688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:46.437688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:46.437688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:46.437688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:46.437688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:46.437688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:46.437688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:46.437688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 367/727] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:46.442892 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfb05e5-fd40-4da1-912f-d0f097d19e13 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:46.461734 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:46.494184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3175f2d0-337f-4cf3-8b75-77caf2d86b55 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:46.531575 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:46.568895 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03524c2b-8735-4449-9d7c-a915106f22f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:46.569507 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03524c2b-8735-4449-9d7c-a915106f22f4 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:46.569783 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03524c2b-8735-4449-9d7c-a915106f22f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:46.570053 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03524c2b-8735-4449-9d7c-a915106f22f4 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:46.570716 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03524c2b-8735-4449-9d7c-a915106f22f4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:46.571018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03524c2b-8735-4449-9d7c-a915106f22f4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:46.573840 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03524c2b-8735-4449-9d7c-a915106f22f4 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:46.601550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 361/728] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:46.607559 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03524c2b-8735-4449-9d7c-a915106f22f4 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:46.626998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfb05e5-fd40-4da1-912f-d0f097d19e13 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:46.661247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a044d42f-2baa-4169-95be-efd187f0700f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:46.661850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a044d42f-2baa-4169-95be-efd187f0700f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:46.662124 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a044d42f-2baa-4169-95be-efd187f0700f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:46.662396 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a044d42f-2baa-4169-95be-efd187f0700f None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:46.663141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a044d42f-2baa-4169-95be-efd187f0700f None admin] RBAC: Adding query filter params (name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:46.663441 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a044d42f-2baa-4169-95be-efd187f0700f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:46.664807 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a044d42f-2baa-4169-95be-efd187f0700f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:46.673784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 368/729] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:27:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:46.675047 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69871e5e-20ca-4842-8138-4688c76fe891 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:46.675333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69871e5e-20ca-4842-8138-4688c76fe891 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:46.675606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69871e5e-20ca-4842-8138-4688c76fe891 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:46.677056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 369/730] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:46.678937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69871e5e-20ca-4842-8138-4688c76fe891 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:46.719834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03524c2b-8735-4449-9d7c-a915106f22f4 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:46.756621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c108db67-4cf9-4d0e-89df-5c8cb150456d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:46.757266 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c108db67-4cf9-4d0e-89df-5c8cb150456d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:46.757546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c108db67-4cf9-4d0e-89df-5c8cb150456d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:46.757928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c108db67-4cf9-4d0e-89df-5c8cb150456d None admin] PATH_INFO: `/v3/users/swift` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:46.764211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:46.765617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c108db67-4cf9-4d0e-89df-5c8cb150456d None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:46.767173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c108db67-4cf9-4d0e-89df-5c8cb150456d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:46.781725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-c108db67-4cf9-4d0e-89df-5c8cb150456d None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 30 09:27:46.781725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:46.781725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:46.781725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:46.781725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:46.781725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:46.781725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:46.781725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:46.781725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:46.781725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:46.781725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:46.781725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:46.781725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:46.781725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:46.781725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:46.781725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:46.782952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:46.782952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:46.782952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:46.782952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:46.782952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:46.782952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:46.782952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:46.782952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:46.782952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:46.782952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:46.782952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:46.782952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:46.782952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:46.782952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:46.782952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:46.782952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:46.782952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:46.782952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:46.783960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:46.783960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:46.783960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:46.783960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:46.783960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:46.783960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:46.783960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:46.783960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:46.783960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 30 09:27:46.783960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:46.786662 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 362/731] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:27:46 2022] GET /identity/v3/users/swift => generated 83 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:46.788088 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad55a41-fa66-4fb0-ab38-4e6c6882b391 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:46.788369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad55a41-fa66-4fb0-ab38-4e6c6882b391 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:46.788636 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad55a41-fa66-4fb0-ab38-4e6c6882b391 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:46.800592 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69871e5e-20ca-4842-8138-4688c76fe891 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:46.810105 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-4ad55a41-fa66-4fb0-ab38-4e6c6882b391 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:46.812069 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4ad55a41-fa66-4fb0-ab38-4e6c6882b391 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:46.839098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c296e32a-a843-4c86-9819-6ea8152a5002 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:46.840789 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c296e32a-a843-4c86-9819-6ea8152a5002 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:46.841275 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c296e32a-a843-4c86-9819-6ea8152a5002 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:46.841777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c296e32a-a843-4c86-9819-6ea8152a5002 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:46.846152 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:46.855134 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c296e32a-a843-4c86-9819-6ea8152a5002 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:46.859243 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c296e32a-a843-4c86-9819-6ea8152a5002 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:46.880598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-c296e32a-a843-4c86-9819-6ea8152a5002 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:46.880598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:46.880598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:46.880598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:46.880598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:46.880598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:46.880598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:46.880598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:46.880598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:46.880598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:46.880598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:46.880598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:46.880598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:46.880598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:46.880598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:46.880598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:46.881894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:46.881894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:46.881894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:46.881894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:46.881894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:46.881894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:46.881894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:46.881894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:46.881894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:46.881894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:46.881894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:46.881894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:46.881894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:46.881894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:46.881894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:46.881894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:46.881894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:46.882855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:46.882855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:46.882855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:46.888977 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 370/732] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:46.888977 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c296e32a-a843-4c86-9819-6ea8152a5002 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:46.897179 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 363/733] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:46.898299 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad55a41-fa66-4fb0-ab38-4e6c6882b391 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:47.029566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad55a41-fa66-4fb0-ab38-4e6c6882b391 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:47.087667 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21094dc-b55b-4c65-863f-03cc8a9764c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:47.088278 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21094dc-b55b-4c65-863f-03cc8a9764c7 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:47.088552 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21094dc-b55b-4c65-863f-03cc8a9764c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:47.088923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21094dc-b55b-4c65-863f-03cc8a9764c7 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:47.089667 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21094dc-b55b-4c65-863f-03cc8a9764c7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:47.090620 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21094dc-b55b-4c65-863f-03cc8a9764c7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:47.091978 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21094dc-b55b-4c65-863f-03cc8a9764c7 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:47.104410 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 364/734] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:47.109796 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21094dc-b55b-4c65-863f-03cc8a9764c7 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:47.110763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c296e32a-a843-4c86-9819-6ea8152a5002 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:47.154771 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:47.162289 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:47.169092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50650f3-bb83-4822-96ec-732f010e675a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:47.169700 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50650f3-bb83-4822-96ec-732f010e675a None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:47.169982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50650f3-bb83-4822-96ec-732f010e675a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:47.170255 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50650f3-bb83-4822-96ec-732f010e675a None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:47.171006 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50650f3-bb83-4822-96ec-732f010e675a None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:47.171315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50650f3-bb83-4822-96ec-732f010e675a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:47.172794 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50650f3-bb83-4822-96ec-732f010e675a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:47.203266 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 371/735] 10.222.0.39 () {56 vars in 1135 bytes} [Fri Sep 30 09:27:46 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:47.204290 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50650f3-bb83-4822-96ec-732f010e675a None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:47.254767 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21094dc-b55b-4c65-863f-03cc8a9764c7 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:47.297989 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0428bf95-eb1e-4f91-8bd4-b4fdc68fc69c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:47.298655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0428bf95-eb1e-4f91-8bd4-b4fdc68fc69c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:47.299012 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0428bf95-eb1e-4f91-8bd4-b4fdc68fc69c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:47.299356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0428bf95-eb1e-4f91-8bd4-b4fdc68fc69c None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:47.306826 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0428bf95-eb1e-4f91-8bd4-b4fdc68fc69c None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:47.308356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0428bf95-eb1e-4f91-8bd4-b4fdc68fc69c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:47.321512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-0428bf95-eb1e-4f91-8bd4-b4fdc68fc69c None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 30 09:27:47.321512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:47.321512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:47.321512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:47.321512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:47.321512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:47.321512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:47.321512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:47.321512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:47.321512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:47.321512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:47.321512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:47.321512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:47.321512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:47.321512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:47.321512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:47.322547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:47.322547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:47.322547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:47.322547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:47.322547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:47.322547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:47.322547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:47.322547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:47.322547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:47.322547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:47.322547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:47.322547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:47.322547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:47.322547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:47.322547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:47.322547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:47.322547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:47.322547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:47.323561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:47.323561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:47.323561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:47.323561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:47.323561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:47.323561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:47.323561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:47.323561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:47.323561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 30 09:27:47.323561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:47.323561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 365/736] 10.222.0.39 () {56 vars in 1117 bytes} [Fri Sep 30 09:27:47 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:47.325225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0428bf95-eb1e-4f91-8bd4-b4fdc68fc69c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:47.333738 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50650f3-bb83-4822-96ec-732f010e675a None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:47.377254 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64587479-9232-4468-96e9-66d1a9026992 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:47.377990 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64587479-9232-4468-96e9-66d1a9026992 None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:47.378150 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64587479-9232-4468-96e9-66d1a9026992 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:47.378371 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64587479-9232-4468-96e9-66d1a9026992 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:47.379064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64587479-9232-4468-96e9-66d1a9026992 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:47.380432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64587479-9232-4468-96e9-66d1a9026992 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:47.422129 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:47.438968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0428bf95-eb1e-4f91-8bd4-b4fdc68fc69c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:47.471932 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a997b4-a76d-4538-a9cc-a98127ccccd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:47.472538 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a997b4-a76d-4538-a9cc-a98127ccccd7 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:47.472819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a997b4-a76d-4538-a9cc-a98127ccccd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:47.473095 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a997b4-a76d-4538-a9cc-a98127ccccd7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:47.487860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 372/737] 10.222.0.39 () {60 vars in 1141 bytes} [Fri Sep 30 09:27:47 2022] POST /identity/v3/services => generated 251 bytes in 284 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:27:47.489504 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64587479-9232-4468-96e9-66d1a9026992 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:47.568184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:47.582279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a997b4-a76d-4538-a9cc-a98127ccccd7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:47.584033 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a997b4-a76d-4538-a9cc-a98127ccccd7 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:47.655800 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64587479-9232-4468-96e9-66d1a9026992 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:47.700172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b945a81b-30a2-4dbc-acdb-c0b2180ea001 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:47.700408 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b945a81b-30a2-4dbc-acdb-c0b2180ea001 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:47.700617 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b945a81b-30a2-4dbc-acdb-c0b2180ea001 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:47.700855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b945a81b-30a2-4dbc-acdb-c0b2180ea001 None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:47.701750 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b945a81b-30a2-4dbc-acdb-c0b2180ea001 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:47.701934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b945a81b-30a2-4dbc-acdb-c0b2180ea001 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:47.703854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b945a81b-30a2-4dbc-acdb-c0b2180ea001 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:47.707522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:47.723476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 366/738] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:27:47.724067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec20d2b-a27c-4b33-b0d2-827becaca7bc None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:47.724360 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec20d2b-a27c-4b33-b0d2-827becaca7bc None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:47.724636 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec20d2b-a27c-4b33-b0d2-827becaca7bc None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:47.726114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 367/739] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:47.728236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 373/740] 10.222.0.39 () {56 vars in 1149 bytes} [Fri Sep 30 09:27:47 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:47.731607 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af88eff7-d657-4d68-8124-09e1c4f6eccb None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:47.731725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec20d2b-a27c-4b33-b0d2-827becaca7bc None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:47.732235 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af88eff7-d657-4d68-8124-09e1c4f6eccb None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:47.732710 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af88eff7-d657-4d68-8124-09e1c4f6eccb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:47.753409 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-af88eff7-d657-4d68-8124-09e1c4f6eccb None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:47.755373 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-af88eff7-d657-4d68-8124-09e1c4f6eccb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:47.845044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 374/741] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:47 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 30 09:27:47.845898 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af88eff7-d657-4d68-8124-09e1c4f6eccb None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:47.853529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec20d2b-a27c-4b33-b0d2-827becaca7bc None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:47.899322 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c1fe6e-15f8-4d8e-8cd5-ff2e2a6673cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:47.899961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c1fe6e-15f8-4d8e-8cd5-ff2e2a6673cd None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:47.900297 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c1fe6e-15f8-4d8e-8cd5-ff2e2a6673cd None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:47.900581 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c1fe6e-15f8-4d8e-8cd5-ff2e2a6673cd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:47.918500 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:47.921946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c1fe6e-15f8-4d8e-8cd5-ff2e2a6673cd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:47.923434 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c1fe6e-15f8-4d8e-8cd5-ff2e2a6673cd None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:47.954302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-c6c1fe6e-15f8-4d8e-8cd5-ff2e2a6673cd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:47.954302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:47.954302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:47.954302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:47.954302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:47.954302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:47.954302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:47.954302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:47.954302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:47.954302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:47.954302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:47.954302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:47.954302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:47.954302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:47.954302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:47.954302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:47.955389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:47.955389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:47.955389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:47.955389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:47.955389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:47.955389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:47.955389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:47.955389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:47.955389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:47.955389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:47.955389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:47.955389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:47.955389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:47.955389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:47.955389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:47.955389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:47.955389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:47.956292 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:47.956292 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:47.956292 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:47.960627 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 368/742] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:47.966721 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c1fe6e-15f8-4d8e-8cd5-ff2e2a6673cd None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:48.046979 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:48.053693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af88eff7-d657-4d68-8124-09e1c4f6eccb None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:48.102089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66203642-8b6e-4d4c-b792-acf70336f29e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:48.102646 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66203642-8b6e-4d4c-b792-acf70336f29e None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:48.102931 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66203642-8b6e-4d4c-b792-acf70336f29e None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:48.103205 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66203642-8b6e-4d4c-b792-acf70336f29e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:48.155510 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c1fe6e-15f8-4d8e-8cd5-ff2e2a6673cd None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:48.196113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bee152d-6ef2-4495-9bc6-64ab9fdda75a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:48.196113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bee152d-6ef2-4495-9bc6-64ab9fdda75a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:48.196113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bee152d-6ef2-4495-9bc6-64ab9fdda75a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:48.196113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bee152d-6ef2-4495-9bc6-64ab9fdda75a None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:48.196517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bee152d-6ef2-4495-9bc6-64ab9fdda75a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:48.197882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bee152d-6ef2-4495-9bc6-64ab9fdda75a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:48.202614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bee152d-6ef2-4495-9bc6-64ab9fdda75a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:48.215152 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 369/743] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:48.217051 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bee152d-6ef2-4495-9bc6-64ab9fdda75a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:48.240075 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66203642-8b6e-4d4c-b792-acf70336f29e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:48.241883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66203642-8b6e-4d4c-b792-acf70336f29e None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:48.279476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:48.362820 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bee152d-6ef2-4495-9bc6-64ab9fdda75a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:48.363373 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 375/744] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:47 2022] GET /identity/v3/auth/tokens => generated 903 bytes in 513 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 30 09:27:48.364490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66203642-8b6e-4d4c-b792-acf70336f29e None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:48.374358 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:48.400592 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f523f1-7953-4c94-833a-554afbfb3672 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:48.401202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f523f1-7953-4c94-833a-554afbfb3672 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:48.401463 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f523f1-7953-4c94-833a-554afbfb3672 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:48.401733 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f523f1-7953-4c94-833a-554afbfb3672 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:48.403458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f523f1-7953-4c94-833a-554afbfb3672 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:48.404270 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f523f1-7953-4c94-833a-554afbfb3672 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:48.467832 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66203642-8b6e-4d4c-b792-acf70336f29e None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:48.473263 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 370/745] 10.222.0.39 () {60 vars in 1141 bytes} [Fri Sep 30 09:27:48 2022] POST /identity/v3/services => generated 240 bytes in 256 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:27:48.481381 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f523f1-7953-4c94-833a-554afbfb3672 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:48.508278 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:48.521457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89159ac-5d5f-4aac-b2bf-294c3605e7ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:48.522125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89159ac-5d5f-4aac-b2bf-294c3605e7ec None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:48.522408 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89159ac-5d5f-4aac-b2bf-294c3605e7ec None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:48.522838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89159ac-5d5f-4aac-b2bf-294c3605e7ec None admin] PATH_INFO: `/v3/projects/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:48.531504 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89159ac-5d5f-4aac-b2bf-294c3605e7ec None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:48.536263 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89159ac-5d5f-4aac-b2bf-294c3605e7ec None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:48.547936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-f89159ac-5d5f-4aac-b2bf-294c3605e7ec None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:48.547936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:48.547936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:48.547936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:48.547936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:48.547936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:48.547936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:48.547936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:48.547936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:48.547936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:48.547936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:48.547936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:48.547936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:48.547936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:48.547936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:48.549119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:48.549119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:48.549119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:48.549119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:48.549119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:48.549119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:48.549119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:48.549119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:48.549119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:48.549119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:48.549119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:48.549119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:48.549119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:48.549119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:48.549119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:48.549119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:48.549119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:48.550068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:48.550068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:48.550068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:48.550068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:48.550068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:48.550068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:48.550068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:48.550068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:48.554651 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 376/746] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:48.556192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c46601c-435f-49ed-af0f-df6e902e2f40 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:48.556391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c46601c-435f-49ed-af0f-df6e902e2f40 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:48.556672 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c46601c-435f-49ed-af0f-df6e902e2f40 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:48.562981 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 377/747] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:48 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:48.564352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c46601c-435f-49ed-af0f-df6e902e2f40 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:48.629651 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f523f1-7953-4c94-833a-554afbfb3672 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:48.673036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0d22e4-0096-4536-8675-a0df0ef8edab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:48.674039 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0d22e4-0096-4536-8675-a0df0ef8edab None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:48.674378 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0d22e4-0096-4536-8675-a0df0ef8edab None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:48.674769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0d22e4-0096-4536-8675-a0df0ef8edab None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:48.686911 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0d22e4-0096-4536-8675-a0df0ef8edab None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:48.688563 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0d22e4-0096-4536-8675-a0df0ef8edab None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:48.695108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c46601c-435f-49ed-af0f-df6e902e2f40 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:48.708737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-8b0d22e4-0096-4536-8675-a0df0ef8edab None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:48.708737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:48.708737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:48.708737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:48.708737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:48.708737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:48.708737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:48.708737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:48.708737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:48.708737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:48.708737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:48.708737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:48.708737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:48.708737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:48.708737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:48.708737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:48.710247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:48.710247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:48.710247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:48.710247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:48.710247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:48.710247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:48.710247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:48.710247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:48.710247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:48.710247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:48.710247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:48.710247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:48.710247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:48.710247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:48.710247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:48.710247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:48.710247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:48.711232 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:48.711232 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:48.711232 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:48.711663 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 371/748] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:48.712818 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bd3627-8b10-4b26-886f-af4f65f7ab33 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:48.713104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bd3627-8b10-4b26-886f-af4f65f7ab33 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:48.713376 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bd3627-8b10-4b26-886f-af4f65f7ab33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:48.733135 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c1bd3627-8b10-4b26-886f-af4f65f7ab33 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:48.734355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c1bd3627-8b10-4b26-886f-af4f65f7ab33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:48.737654 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:48.739583 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82926a9-7004-412e-a91d-fa3bdf01f104 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:48.739813 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82926a9-7004-412e-a91d-fa3bdf01f104 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:48.740045 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82926a9-7004-412e-a91d-fa3bdf01f104 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:48.740323 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82926a9-7004-412e-a91d-fa3bdf01f104 None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:48.741017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82926a9-7004-412e-a91d-fa3bdf01f104 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:48.741326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82926a9-7004-412e-a91d-fa3bdf01f104 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:48.743689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82926a9-7004-412e-a91d-fa3bdf01f104 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:48.757149 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 378/749] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:48.762729 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82926a9-7004-412e-a91d-fa3bdf01f104 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:48.818779 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 372/750] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:48 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:27:48.819888 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bd3627-8b10-4b26-886f-af4f65f7ab33 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:48.820733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:48.871670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82926a9-7004-412e-a91d-fa3bdf01f104 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:48.936148 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bd3627-8b10-4b26-886f-af4f65f7ab33 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:48.945057 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a94a9e4-d806-4c22-a9e8-ba9310ffab3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:48.945713 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a94a9e4-d806-4c22-a9e8-ba9310ffab3a None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:48.945994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a94a9e4-d806-4c22-a9e8-ba9310ffab3a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:48.946270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a94a9e4-d806-4c22-a9e8-ba9310ffab3a None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:48.946945 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a94a9e4-d806-4c22-a9e8-ba9310ffab3a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:48.947252 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a94a9e4-d806-4c22-a9e8-ba9310ffab3a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:48.948810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a94a9e4-d806-4c22-a9e8-ba9310ffab3a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:48.970746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 379/751] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:48.977488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a94a9e4-d806-4c22-a9e8-ba9310ffab3a None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:48.978719 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8341bd-27f7-45f4-a69f-599d902f7c78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:48.979614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8341bd-27f7-45f4-a69f-599d902f7c78 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:48.980090 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8341bd-27f7-45f4-a69f-599d902f7c78 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:48.980619 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8341bd-27f7-45f4-a69f-599d902f7c78 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:48.981651 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8341bd-27f7-45f4-a69f-599d902f7c78 None admin] RBAC: Adding query filter params (role.id=e874c311084c4c9fb7691bce461d319b, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=0e5ad921e6fa41c297696ec7279f0fc4) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:48.982256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8341bd-27f7-45f4-a69f-599d902f7c78 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:48.983971 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8341bd-27f7-45f4-a69f-599d902f7c78 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:48.990239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 373/752] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:27:48 2022] GET /identity/v3/role_assignments?user.id=0e5ad921e6fa41c297696ec7279f0fc4&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=e874c311084c4c9fb7691bce461d319b => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:49.129417 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8341bd-27f7-45f4-a69f-599d902f7c78 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:49.140625 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:49.147188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a94a9e4-d806-4c22-a9e8-ba9310ffab3a None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:49.158962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:49.221994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91014332-2efe-4a8d-a8d4-2e6a211e251f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:49.222660 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91014332-2efe-4a8d-a8d4-2e6a211e251f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:49.222946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91014332-2efe-4a8d-a8d4-2e6a211e251f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:49.223215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91014332-2efe-4a8d-a8d4-2e6a211e251f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:49.235062 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91014332-2efe-4a8d-a8d4-2e6a211e251f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:49.236646 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91014332-2efe-4a8d-a8d4-2e6a211e251f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:49.259329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-91014332-2efe-4a8d-a8d4-2e6a211e251f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:49.259329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:49.259329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:49.259329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:49.259329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:49.259329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:49.259329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:49.259329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:49.259329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:49.259329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:49.259329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:49.259329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:49.259329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:49.259329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:49.259329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:49.260301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:49.260301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:49.260301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:49.260301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:49.260301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:49.260301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:49.260301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:49.260301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:49.260301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:49.260301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:49.260301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:49.260301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:49.260301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:49.260301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:49.260301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:49.260301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:49.260301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:49.261229 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:49.261229 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:49.261229 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:49.261229 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:49.261229 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:49.261229 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:49.261229 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:49.261229 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:49.261229 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 380/753] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:49.268207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91014332-2efe-4a8d-a8d4-2e6a211e251f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:49.291759 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8341bd-27f7-45f4-a69f-599d902f7c78 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:49.341500 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa037294-cff4-4f31-8604-b3107d7f0e57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:49.346887 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa037294-cff4-4f31-8604-b3107d7f0e57 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:49.347460 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa037294-cff4-4f31-8604-b3107d7f0e57 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:49.347955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa037294-cff4-4f31-8604-b3107d7f0e57 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:49.366369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa037294-cff4-4f31-8604-b3107d7f0e57 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:49.379056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-aa037294-cff4-4f31-8604-b3107d7f0e57 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:27:49.379056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:49.379056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:49.379056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:49.379056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:49.379056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:49.379056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:49.379056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:49.379056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:49.379056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:49.379056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:49.379056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:49.379056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:49.379056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:49.379056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:49.379056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:49.380060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:49.380060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:49.380060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:49.380060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:49.380060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:49.380060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:49.380060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:49.380060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:49.380060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:49.380060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:49.380060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:49.380060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:49.380060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:49.380060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:49.380060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:49.380060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:49.380925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:49.380925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:49.380925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:49.380925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:49.380925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:49.380925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:49.380925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:49.380925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:49.380925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:49.380925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:49.380925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:27:49.380925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:49.384511 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 374/754] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:27:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:49.392349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa037294-cff4-4f31-8604-b3107d7f0e57 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:49.449470 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91014332-2efe-4a8d-a8d4-2e6a211e251f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:49.489995 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:49.505192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ef1d7e-594c-4f3d-95e2-3e5afc11baac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:49.505858 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ef1d7e-594c-4f3d-95e2-3e5afc11baac None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:49.506346 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ef1d7e-594c-4f3d-95e2-3e5afc11baac None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:49.510637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ef1d7e-594c-4f3d-95e2-3e5afc11baac None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:49.512191 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ef1d7e-594c-4f3d-95e2-3e5afc11baac None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:49.512512 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ef1d7e-594c-4f3d-95e2-3e5afc11baac None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:49.513981 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ef1d7e-594c-4f3d-95e2-3e5afc11baac None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:49.539861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 381/755] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:49.542189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ef1d7e-594c-4f3d-95e2-3e5afc11baac None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:49.570201 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa037294-cff4-4f31-8604-b3107d7f0e57 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:49.598961 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:49.606177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335b2ce3-f693-4ee5-991e-fcc0dda1482f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:49.606819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335b2ce3-f693-4ee5-991e-fcc0dda1482f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:49.607105 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335b2ce3-f693-4ee5-991e-fcc0dda1482f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:49.607383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335b2ce3-f693-4ee5-991e-fcc0dda1482f None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:49.608113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335b2ce3-f693-4ee5-991e-fcc0dda1482f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:49.608423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335b2ce3-f693-4ee5-991e-fcc0dda1482f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:49.609820 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335b2ce3-f693-4ee5-991e-fcc0dda1482f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:49.619242 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 375/756] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:27:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:49.625264 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335b2ce3-f693-4ee5-991e-fcc0dda1482f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:49.732044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335b2ce3-f693-4ee5-991e-fcc0dda1482f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:49.758438 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:49.760973 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ef1d7e-594c-4f3d-95e2-3e5afc11baac None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:49.767543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a187acc9-4965-4935-b12f-4ed8287a7838 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:49.768181 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a187acc9-4965-4935-b12f-4ed8287a7838 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:49.768465 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a187acc9-4965-4935-b12f-4ed8287a7838 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:49.768744 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a187acc9-4965-4935-b12f-4ed8287a7838 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:49.780249 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a187acc9-4965-4935-b12f-4ed8287a7838 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:49.781704 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a187acc9-4965-4935-b12f-4ed8287a7838 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:49.796800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-a187acc9-4965-4935-b12f-4ed8287a7838 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:49.796800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:49.796800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:49.796800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:49.796800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:49.796800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:49.796800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:49.796800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:49.796800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:49.796800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:49.796800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:49.796800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:49.796800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:49.796800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:49.796800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:49.796800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:49.798168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:49.798168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:49.798168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:49.798168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:49.798168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:49.798168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:49.798168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:49.798168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:49.798168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:49.798168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:49.798168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:49.798168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:49.798168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:49.798168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:49.798168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:49.798168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:49.798168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:49.799609 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:49.799609 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:49.799609 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:49.799609 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 376/757] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:49.803705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a187acc9-4965-4935-b12f-4ed8287a7838 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:49.836522 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43afa34c-ff4e-4134-a639-150f2c89463f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:49.837239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43afa34c-ff4e-4134-a639-150f2c89463f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:49.837518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43afa34c-ff4e-4134-a639-150f2c89463f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:49.838034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43afa34c-ff4e-4134-a639-150f2c89463f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:49.838633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43afa34c-ff4e-4134-a639-150f2c89463f None admin] RBAC: Adding query filter params (role.id=144dd855f74945548ea65db37acda1c1, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=7d6e0545a40a436d8c63820b795551a9) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:49.838932 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43afa34c-ff4e-4134-a639-150f2c89463f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:49.840881 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43afa34c-ff4e-4134-a639-150f2c89463f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:49.846787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 382/758] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:27:49 2022] GET /identity/v3/role_assignments?user.id=7d6e0545a40a436d8c63820b795551a9&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=144dd855f74945548ea65db37acda1c1 => generated 259 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:49.961236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a187acc9-4965-4935-b12f-4ed8287a7838 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:50.012678 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fbfe2a-9532-423b-ad5b-68fb4b25bdd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:50.013231 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fbfe2a-9532-423b-ad5b-68fb4b25bdd0 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:50.013513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fbfe2a-9532-423b-ad5b-68fb4b25bdd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:50.013792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fbfe2a-9532-423b-ad5b-68fb4b25bdd0 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:50.014445 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fbfe2a-9532-423b-ad5b-68fb4b25bdd0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:50.017966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fbfe2a-9532-423b-ad5b-68fb4b25bdd0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:50.019486 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fbfe2a-9532-423b-ad5b-68fb4b25bdd0 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:50.036038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 377/759] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:50.041819 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43afa34c-ff4e-4134-a639-150f2c89463f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:50.052258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef96f058-b99c-4bbc-9fa3-f79990a87ffb None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:50.052545 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef96f058-b99c-4bbc-9fa3-f79990a87ffb None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:50.052776 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef96f058-b99c-4bbc-9fa3-f79990a87ffb None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:50.054363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 378/760] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:50.059474 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badff726-ad04-4541-8004-826459fd675f None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:50.062171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badff726-ad04-4541-8004-826459fd675f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:50.062171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badff726-ad04-4541-8004-826459fd675f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:50.080209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-badff726-ad04-4541-8004-826459fd675f None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:50.081401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-badff726-ad04-4541-8004-826459fd675f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:50.178209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:50.186981 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:50.188099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43afa34c-ff4e-4134-a639-150f2c89463f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:50.229153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec0aca7-dbec-4892-9d3a-8ea8f8cba01e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:50.229799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec0aca7-dbec-4892-9d3a-8ea8f8cba01e None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:50.230106 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec0aca7-dbec-4892-9d3a-8ea8f8cba01e None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:50.230387 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec0aca7-dbec-4892-9d3a-8ea8f8cba01e None admin] PATH_INFO: `/v3/users/nova` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:50.238498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec0aca7-dbec-4892-9d3a-8ea8f8cba01e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:50.240123 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec0aca7-dbec-4892-9d3a-8ea8f8cba01e None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:50.252189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-7ec0aca7-dbec-4892-9d3a-8ea8f8cba01e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 30 09:27:50.252189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:50.252189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:50.252189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:50.252189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:50.252189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:50.252189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:50.252189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:50.252189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:50.252189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:50.252189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:50.252189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:50.252189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:50.252189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:50.252189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:50.252189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:50.253174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:50.253174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:50.253174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:50.253174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:50.253174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:50.253174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:50.253174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:50.253174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:50.253174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:50.253174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:50.253174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:50.253174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:50.253174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:50.253174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:50.253174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:50.253174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:50.253174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:50.253174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:50.254153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:50.254153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:50.254153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:50.254153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:50.254153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:50.254153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:50.254153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:50.254153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:50.254153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 30 09:27:50.254153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:50.254153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 383/761] 10.222.0.39 () {56 vars in 1093 bytes} [Fri Sep 30 09:27:50 2022] GET /identity/v3/users/nova => generated 82 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:50.259353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec0aca7-dbec-4892-9d3a-8ea8f8cba01e None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:50.312266 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 379/762] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:50 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 30 09:27:50.387080 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec0aca7-dbec-4892-9d3a-8ea8f8cba01e None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:50.439887 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519c5592-49c9-4d67-a953-06b6dc16540d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:50.440502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519c5592-49c9-4d67-a953-06b6dc16540d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:50.440780 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519c5592-49c9-4d67-a953-06b6dc16540d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:50.441053 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519c5592-49c9-4d67-a953-06b6dc16540d None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:50.441777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519c5592-49c9-4d67-a953-06b6dc16540d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:50.442082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519c5592-49c9-4d67-a953-06b6dc16540d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:50.443621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519c5592-49c9-4d67-a953-06b6dc16540d None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:50.471799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 384/763] 10.222.0.39 () {56 vars in 1133 bytes} [Fri Sep 30 09:27:50 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:50.479685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badff726-ad04-4541-8004-826459fd675f None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:50.628102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badff726-ad04-4541-8004-826459fd675f None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:50.676972 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca84e7b8-4c06-45fe-9853-bc32523edcaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:50.677608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca84e7b8-4c06-45fe-9853-bc32523edcaf None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:50.677888 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca84e7b8-4c06-45fe-9853-bc32523edcaf None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:50.678160 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca84e7b8-4c06-45fe-9853-bc32523edcaf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:50.698392 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca84e7b8-4c06-45fe-9853-bc32523edcaf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:50.700448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca84e7b8-4c06-45fe-9853-bc32523edcaf None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:50.718496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-ca84e7b8-4c06-45fe-9853-bc32523edcaf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:50.718496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:50.718496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:50.718496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:50.718496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:50.718496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:50.718496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:50.718496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:50.718496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:50.718496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:50.718496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:50.718496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:50.718496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:50.718496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:50.718496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:50.718496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:50.720345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:50.720345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:50.720345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:50.720345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:50.720345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:50.720345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:50.720345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:50.720345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:50.720345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:50.720345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:50.720345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:50.720345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:50.720345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:50.720345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:50.720345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:50.720345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:50.720345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:50.722120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:50.722120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:50.722120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:50.722120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 380/764] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:50.732267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519c5592-49c9-4d67-a953-06b6dc16540d None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:50.758685 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:50.932606 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca84e7b8-4c06-45fe-9853-bc32523edcaf None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:50.972878 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:50.979029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519c5592-49c9-4d67-a953-06b6dc16540d None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:51.038796 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a9a386-150a-43ee-8ee3-52618a4f5793 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:51.039880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a9a386-150a-43ee-8ee3-52618a4f5793 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:51.041328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a9a386-150a-43ee-8ee3-52618a4f5793 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:51.041961 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a9a386-150a-43ee-8ee3-52618a4f5793 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:51.043874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a9a386-150a-43ee-8ee3-52618a4f5793 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:51.044374 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a9a386-150a-43ee-8ee3-52618a4f5793 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:51.046108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a9a386-150a-43ee-8ee3-52618a4f5793 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:51.069396 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 385/765] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 338 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:51.075425 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a9a386-150a-43ee-8ee3-52618a4f5793 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:51.119943 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca84e7b8-4c06-45fe-9853-bc32523edcaf None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:51.177141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:51.193111 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead81baf-c03f-4907-b7f8-4e8368dafe2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:51.194066 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead81baf-c03f-4907-b7f8-4e8368dafe2b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:51.195018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead81baf-c03f-4907-b7f8-4e8368dafe2b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:51.195304 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead81baf-c03f-4907-b7f8-4e8368dafe2b None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:51.195935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead81baf-c03f-4907-b7f8-4e8368dafe2b None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:51.197392 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead81baf-c03f-4907-b7f8-4e8368dafe2b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:51.218624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-ead81baf-c03f-4907-b7f8-4e8368dafe2b None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 30 09:27:51.218624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:51.218624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:27:51.218624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:27:51.218624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:27:51.218624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:27:51.218624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:27:51.218624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:27:51.218624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 30 09:27:51.218624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:51.218624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:27:51.218624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:51.218624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:51.218624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:51.218624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:51.218624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:51.218624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:51.219713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:51.219713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:51.219713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:51.219713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:51.219713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:51.219713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:51.219713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:27:51.219713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:27:51.219713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:27:51.219713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:27:51.219713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:51.219713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:51.219713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:51.219713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:51.219713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:51.219713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:51.220589 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:51.220589 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:51.220589 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:51.220589 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:51.220589 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:51.220589 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:51.220589 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:51.220589 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:51.220589 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:51.220589 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:51.220589 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:27:51.220589 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:27:51.220589 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 30 09:27:51.220589 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:51.220589 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 381/766] 10.222.0.39 () {56 vars in 1129 bytes} [Fri Sep 30 09:27:50 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:51.225139 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead81baf-c03f-4907-b7f8-4e8368dafe2b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:51.244283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a9a386-150a-43ee-8ee3-52618a4f5793 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:51.288267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e415cd-502b-46c1-9946-ca4bdf14b6fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:51.290192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e415cd-502b-46c1-9946-ca4bdf14b6fd None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:51.291949 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e415cd-502b-46c1-9946-ca4bdf14b6fd None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:51.292462 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e415cd-502b-46c1-9946-ca4bdf14b6fd None admin] PATH_INFO: `/v3/projects/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:51.302231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e415cd-502b-46c1-9946-ca4bdf14b6fd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:51.303726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e415cd-502b-46c1-9946-ca4bdf14b6fd None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:51.315579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-c6e415cd-502b-46c1-9946-ca4bdf14b6fd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:51.315579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:51.315579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:51.315579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:51.315579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:51.315579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:51.315579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:51.315579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:51.315579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:51.315579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:51.315579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:51.315579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:51.315579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:51.315579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:51.315579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:51.316542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:51.316542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:51.316542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:51.316542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:51.316542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:51.316542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:51.316542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:51.316542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:51.316542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:51.316542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:51.316542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:51.316542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:51.316542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:51.316542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:51.316542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:51.316542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:51.316542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:51.317455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:51.317455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:51.317455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:51.317455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:51.317455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:51.317455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:51.317455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:51.317455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:51.317455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 386/767] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:51.318417 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbef2d01-7ea4-4acf-8295-aee8df9f4424 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:51.318417 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbef2d01-7ea4-4acf-8295-aee8df9f4424 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:51.318604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbef2d01-7ea4-4acf-8295-aee8df9f4424 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:51.324042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 387/768] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:51 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:51.325104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbef2d01-7ea4-4acf-8295-aee8df9f4424 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:51.368514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:51.392118 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead81baf-c03f-4907-b7f8-4e8368dafe2b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:51.429969 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9f0b80-5863-4984-b0cb-8f07355e7cb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:51.430573 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9f0b80-5863-4984-b0cb-8f07355e7cb1 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:51.430896 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9f0b80-5863-4984-b0cb-8f07355e7cb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:51.431348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9f0b80-5863-4984-b0cb-8f07355e7cb1 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:51.431766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9f0b80-5863-4984-b0cb-8f07355e7cb1 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:51.432062 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9f0b80-5863-4984-b0cb-8f07355e7cb1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:51.434003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9f0b80-5863-4984-b0cb-8f07355e7cb1 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:51.439321 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 382/769] 10.222.0.39 () {56 vars in 1124 bytes} [Fri Sep 30 09:27:51 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:51.440473 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fe8ba8-4533-466e-9ada-48edac8a29f0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:51.440753 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fe8ba8-4533-466e-9ada-48edac8a29f0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:51.441021 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fe8ba8-4533-466e-9ada-48edac8a29f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:51.445098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbef2d01-7ea4-4acf-8295-aee8df9f4424 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:51.459531 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-75fe8ba8-4533-466e-9ada-48edac8a29f0 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:51.460706 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-75fe8ba8-4533-466e-9ada-48edac8a29f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:51.486399 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:51.498157 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece963f6-8ca3-4f69-a2ca-b88bfe7c4eb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:51.502115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece963f6-8ca3-4f69-a2ca-b88bfe7c4eb8 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:51.502115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece963f6-8ca3-4f69-a2ca-b88bfe7c4eb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:51.502115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece963f6-8ca3-4f69-a2ca-b88bfe7c4eb8 None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:51.502115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece963f6-8ca3-4f69-a2ca-b88bfe7c4eb8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:51.502115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece963f6-8ca3-4f69-a2ca-b88bfe7c4eb8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:51.503838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece963f6-8ca3-4f69-a2ca-b88bfe7c4eb8 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:51.519824 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 388/770] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:51.521434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece963f6-8ca3-4f69-a2ca-b88bfe7c4eb8 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:51.542758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 383/771] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:51 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:27:51.544032 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fe8ba8-4533-466e-9ada-48edac8a29f0 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:51.643203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece963f6-8ca3-4f69-a2ca-b88bfe7c4eb8 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:51.681725 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916f320d-5d70-44a2-a9ad-feeab23e9c0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:51.682885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916f320d-5d70-44a2-a9ad-feeab23e9c0f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:51.684030 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916f320d-5d70-44a2-a9ad-feeab23e9c0f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:51.684467 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916f320d-5d70-44a2-a9ad-feeab23e9c0f None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:51.685247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916f320d-5d70-44a2-a9ad-feeab23e9c0f None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:51.685699 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916f320d-5d70-44a2-a9ad-feeab23e9c0f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:51.687690 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916f320d-5d70-44a2-a9ad-feeab23e9c0f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:51.693737 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 389/772] 10.222.0.39 () {56 vars in 1124 bytes} [Fri Sep 30 09:27:51 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:51.697317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab2047d-7f2d-478c-b294-c5a51ec024be None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:51.697788 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab2047d-7f2d-478c-b294-c5a51ec024be None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:51.698222 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab2047d-7f2d-478c-b294-c5a51ec024be None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:51.700548 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 390/773] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:51.701396 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab2047d-7f2d-478c-b294-c5a51ec024be None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:51.711752 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:51.735069 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fe8ba8-4533-466e-9ada-48edac8a29f0 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:51.762988 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:51.770471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36275ce2-65f2-4422-aa78-376370c77051 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:51.771273 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36275ce2-65f2-4422-aa78-376370c77051 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:51.771523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36275ce2-65f2-4422-aa78-376370c77051 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:51.771793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36275ce2-65f2-4422-aa78-376370c77051 None admin] PATH_INFO: `/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/7ff68fb87700423ab53c90520a6bd3b9/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:51.773671 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36275ce2-65f2-4422-aa78-376370c77051 None admin] RBAC: Authorizing `identity:create_grant(project_id=c3802ced1fce4997a016d41eca843ef8, user_id=7ff68fb87700423ab53c90520a6bd3b9, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:51.775141 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36275ce2-65f2-4422-aa78-376370c77051 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:51.823755 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab2047d-7f2d-478c-b294-c5a51ec024be None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:51.860731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 384/774] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:27:51 2022] PUT /identity/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/7ff68fb87700423ab53c90520a6bd3b9/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:27:51.861423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64cb438b-cf3b-4133-a350-8986862b4933 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:51.862063 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9e19e0-d706-48d6-8067-fcd25d4c1289 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:51.862447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9e19e0-d706-48d6-8067-fcd25d4c1289 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:51.862551 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64cb438b-cf3b-4133-a350-8986862b4933 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:51.862756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9e19e0-d706-48d6-8067-fcd25d4c1289 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:51.863127 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64cb438b-cf3b-4133-a350-8986862b4933 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:51.863621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64cb438b-cf3b-4133-a350-8986862b4933 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:51.864481 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64cb438b-cf3b-4133-a350-8986862b4933 None admin] RBAC: Adding query filter params (interface=public, service_id=91c735d14bfc4f8c9be58253fb98609f, region_id=RegionOne) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:51.864973 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64cb438b-cf3b-4133-a350-8986862b4933 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:51.866884 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64cb438b-cf3b-4133-a350-8986862b4933 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:51.876791 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 391/775] 10.222.0.39 () {56 vars in 1251 bytes} [Fri Sep 30 09:27:51 2022] GET /identity/v3/endpoints?service_id=91c735d14bfc4f8c9be58253fb98609f&interface=public®ion_id=RegionOne => generated 194 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:51.877910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64cb438b-cf3b-4133-a350-8986862b4933 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:51.888189 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ff9e19e0-d706-48d6-8067-fcd25d4c1289 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:51.888189 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ff9e19e0-d706-48d6-8067-fcd25d4c1289 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:51.974955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 385/776] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:51 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:27:51.975867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9e19e0-d706-48d6-8067-fcd25d4c1289 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:52.036687 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64cb438b-cf3b-4133-a350-8986862b4933 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:52.056834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:52.079923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7373fb-36e1-44cd-be15-e428db1670ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:52.080562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7373fb-36e1-44cd-be15-e428db1670ac None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:52.080927 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7373fb-36e1-44cd-be15-e428db1670ac None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:52.081160 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7373fb-36e1-44cd-be15-e428db1670ac None admin] PATH_INFO: `/v3/services/network` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:52.081803 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7373fb-36e1-44cd-be15-e428db1670ac None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:52.085421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7373fb-36e1-44cd-be15-e428db1670ac None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:52.093159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:52.095802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-7b7373fb-36e1-44cd-be15-e428db1670ac None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 30 09:27:52.095802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:52.095802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:27:52.095802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:27:52.095802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:27:52.095802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:27:52.095802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:27:52.095802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:27:52.095802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 30 09:27:52.095802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:52.095802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:27:52.095802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:52.095802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:52.095802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:52.095802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:52.095802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:52.095802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:52.098312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:52.098312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:52.098312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:52.098312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:52.098312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:52.098312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:52.098312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:27:52.098312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:27:52.098312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:27:52.098312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:27:52.098312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:52.098312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:52.098312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:52.098312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:52.098312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:52.098312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:52.099274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:52.099274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:52.099274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:52.099274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:52.099274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:52.099274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:52.099274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:52.099274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:52.099274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:52.099274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:52.099274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:27:52.099274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:27:52.099274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 30 09:27:52.099274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:52.099274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 392/777] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:51 2022] GET /identity/v3/services/network => generated 88 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:52.107590 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7373fb-36e1-44cd-be15-e428db1670ac None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:52.135650 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9e19e0-d706-48d6-8067-fcd25d4c1289 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:52.178730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33b44c3-1543-4153-bd02-5bb3fda2c2cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:52.179813 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33b44c3-1543-4153-bd02-5bb3fda2c2cd None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:52.180241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33b44c3-1543-4153-bd02-5bb3fda2c2cd None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:52.180655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33b44c3-1543-4153-bd02-5bb3fda2c2cd None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:52.181499 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33b44c3-1543-4153-bd02-5bb3fda2c2cd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:52.183264 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33b44c3-1543-4153-bd02-5bb3fda2c2cd None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:52.190106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 386/778] 10.222.0.39 () {56 vars in 1087 bytes} [Fri Sep 30 09:27:51 2022] GET /identity/v3/services => generated 772 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:52.247431 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7373fb-36e1-44cd-be15-e428db1670ac None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:52.290283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d345d2-9a94-4d3d-b78b-e73ec87ccd12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:52.290719 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33b44c3-1543-4153-bd02-5bb3fda2c2cd None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:52.290929 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d345d2-9a94-4d3d-b78b-e73ec87ccd12 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:52.291206 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d345d2-9a94-4d3d-b78b-e73ec87ccd12 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:52.291473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d345d2-9a94-4d3d-b78b-e73ec87ccd12 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:52.292104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d345d2-9a94-4d3d-b78b-e73ec87ccd12 None admin] RBAC: Adding query filter params (name=network) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:52.292406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d345d2-9a94-4d3d-b78b-e73ec87ccd12 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:52.294369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d345d2-9a94-4d3d-b78b-e73ec87ccd12 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:52.300447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 393/779] 10.222.0.39 () {56 vars in 1112 bytes} [Fri Sep 30 09:27:52 2022] GET /identity/v3/services?name=network => generated 124 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:52.306509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d345d2-9a94-4d3d-b78b-e73ec87ccd12 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:52.375933 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:52.437660 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d345d2-9a94-4d3d-b78b-e73ec87ccd12 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:52.451466 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33b44c3-1543-4153-bd02-5bb3fda2c2cd None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:52.482919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f22a1bd-19ad-424d-b9cd-7e08253dbaf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:52.483634 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f22a1bd-19ad-424d-b9cd-7e08253dbaf2 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:52.483831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f22a1bd-19ad-424d-b9cd-7e08253dbaf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:52.484081 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f22a1bd-19ad-424d-b9cd-7e08253dbaf2 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:52.484712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f22a1bd-19ad-424d-b9cd-7e08253dbaf2 None admin] RBAC: Adding query filter params (type=network) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:52.485019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f22a1bd-19ad-424d-b9cd-7e08253dbaf2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:52.486394 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f22a1bd-19ad-424d-b9cd-7e08253dbaf2 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:52.488276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:52.491851 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 394/780] 10.222.0.39 () {56 vars in 1112 bytes} [Fri Sep 30 09:27:52 2022] GET /identity/v3/services?type=network => generated 350 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:52.496795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a0542a-9644-4125-ab32-4cd8bc072e11 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:52.496795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a0542a-9644-4125-ab32-4cd8bc072e11 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:52.496795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a0542a-9644-4125-ab32-4cd8bc072e11 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:52.496795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 395/781] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:52 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:52.503054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a0542a-9644-4125-ab32-4cd8bc072e11 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:52.503150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2728ccb-a6b6-4f6f-9b21-a9a978aad760 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:52.503150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2728ccb-a6b6-4f6f-9b21-a9a978aad760 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:52.503150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2728ccb-a6b6-4f6f-9b21-a9a978aad760 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:52.503150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2728ccb-a6b6-4f6f-9b21-a9a978aad760 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:52.513037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2728ccb-a6b6-4f6f-9b21-a9a978aad760 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:52.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-a2728ccb-a6b6-4f6f-9b21-a9a978aad760 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:27:52.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:52.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:52.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:52.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:52.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:52.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:52.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:52.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:52.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:52.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:52.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:52.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:52.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:52.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:52.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:52.519157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:52.519157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:52.519157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:52.519157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:52.519157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:52.519157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:52.519157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:52.519157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:52.519157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:52.519157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:52.519157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:52.519157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:52.519157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:52.519157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:52.519157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:52.519157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:52.520027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:52.520027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:52.520027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:52.520027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:52.520027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:52.520027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:52.520027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:52.520027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:52.520027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:52.520027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:52.520027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:27:52.520027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:52.524261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 387/782] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:27:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:52.526052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f3ca7c-10f2-4355-be41-1d76f3f56d4a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:52.526339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f3ca7c-10f2-4355-be41-1d76f3f56d4a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:52.526638 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f3ca7c-10f2-4355-be41-1d76f3f56d4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:52.545458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f8f3ca7c-10f2-4355-be41-1d76f3f56d4a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:52.546760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f8f3ca7c-10f2-4355-be41-1d76f3f56d4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:52.609506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a0542a-9644-4125-ab32-4cd8bc072e11 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:52.618801 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 388/783] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:52 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:27:52.619676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:52.620247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f3ca7c-10f2-4355-be41-1d76f3f56d4a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:52.671428 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd7894e-0451-43cb-82de-76227d60bd0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:52.672046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd7894e-0451-43cb-82de-76227d60bd0a None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:52.672325 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd7894e-0451-43cb-82de-76227d60bd0a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:52.672596 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd7894e-0451-43cb-82de-76227d60bd0a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:52.673296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd7894e-0451-43cb-82de-76227d60bd0a None admin] RBAC: Adding query filter params (interface=public, service_id=9372c1ff171d41ad93dbde5164df22a4, region_id=RegionOne) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:52.673604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd7894e-0451-43cb-82de-76227d60bd0a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:52.675016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd7894e-0451-43cb-82de-76227d60bd0a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:52.688937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 396/784] 10.222.0.39 () {56 vars in 1251 bytes} [Fri Sep 30 09:27:52 2022] GET /identity/v3/endpoints?service_id=9372c1ff171d41ad93dbde5164df22a4&interface=public®ion_id=RegionOne => generated 194 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:52.694347 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd7894e-0451-43cb-82de-76227d60bd0a None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:52.743114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f3ca7c-10f2-4355-be41-1d76f3f56d4a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:52.769259 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:52.784982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe24a43-de98-4e8f-b109-f1ae16cff576 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:52.785962 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe24a43-de98-4e8f-b109-f1ae16cff576 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:52.786465 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe24a43-de98-4e8f-b109-f1ae16cff576 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:52.787088 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe24a43-de98-4e8f-b109-f1ae16cff576 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:52.788104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe24a43-de98-4e8f-b109-f1ae16cff576 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:52.788617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe24a43-de98-4e8f-b109-f1ae16cff576 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:52.790307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe24a43-de98-4e8f-b109-f1ae16cff576 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:52.796400 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd7894e-0451-43cb-82de-76227d60bd0a None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:52.804660 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 389/785] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:27:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:52.807009 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe24a43-de98-4e8f-b109-f1ae16cff576 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:52.833293 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5f70ac-2138-40a8-94e3-865db0d809b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:52.833904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5f70ac-2138-40a8-94e3-865db0d809b8 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:52.834182 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5f70ac-2138-40a8-94e3-865db0d809b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:52.834458 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5f70ac-2138-40a8-94e3-865db0d809b8 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:52.835293 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5f70ac-2138-40a8-94e3-865db0d809b8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:52.836520 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5f70ac-2138-40a8-94e3-865db0d809b8 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:52.842231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 397/786] 10.222.0.39 () {56 vars in 1087 bytes} [Fri Sep 30 09:27:52 2022] GET /identity/v3/services => generated 772 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:52.922463 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe24a43-de98-4e8f-b109-f1ae16cff576 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:52.959541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7002634-3a2e-4b25-b397-18d1f66476e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:52.960666 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7002634-3a2e-4b25-b397-18d1f66476e3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:52.961154 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7002634-3a2e-4b25-b397-18d1f66476e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:52.961672 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7002634-3a2e-4b25-b397-18d1f66476e3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:52.973987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7002634-3a2e-4b25-b397-18d1f66476e3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:52.976017 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7002634-3a2e-4b25-b397-18d1f66476e3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:52.980633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5f70ac-2138-40a8-94e3-865db0d809b8 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:52.995478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-b7002634-3a2e-4b25-b397-18d1f66476e3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:52.995478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:52.995478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:52.995478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:52.995478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:52.995478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:52.995478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:52.995478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:52.995478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:52.995478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:52.995478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:52.995478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:52.995478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:52.995478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:52.995478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:52.995478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:52.997038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:52.997038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:52.997038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:52.997038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:52.997038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:52.997038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:52.997038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:52.997038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:52.997038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:52.997038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:52.997038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:52.997038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:52.997038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:52.997038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:52.997038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:52.997038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:52.997038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:52.997954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:52.997954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:52.997954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:53.005046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 390/787] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:53.007371 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7002634-3a2e-4b25-b397-18d1f66476e3 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:53.053380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:53.094720 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:53.171281 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5f70ac-2138-40a8-94e3-865db0d809b8 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:53.189914 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7002634-3a2e-4b25-b397-18d1f66476e3 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:53.207974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627c14d7-cd9c-495e-aaed-65e01b906348 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:53.208521 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627c14d7-cd9c-495e-aaed-65e01b906348 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:53.208802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627c14d7-cd9c-495e-aaed-65e01b906348 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:53.209073 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627c14d7-cd9c-495e-aaed-65e01b906348 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:53.220863 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627c14d7-cd9c-495e-aaed-65e01b906348 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:53.225702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-627c14d7-cd9c-495e-aaed-65e01b906348 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:53.225702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:53.225702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:53.225702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:53.225702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:53.225702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:53.225702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:53.225702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:53.225702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:53.225702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:53.225702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:53.225702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:53.225702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:53.225702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:53.225702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:53.225702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:53.226919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:53.226919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:53.226919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:53.226919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:53.226919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:53.226919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:53.226919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:53.226919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:53.226919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:53.226919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:53.226919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:53.226919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:53.226919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:53.226919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:53.226919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:53.226919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:53.227994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:53.227994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:53.227994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:53.227994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:53.227994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:53.227994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:53.227994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:53.227994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:53.227994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:53.227994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:53.227994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:53.227994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:53.227994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 398/788] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:53.233869 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627c14d7-cd9c-495e-aaed-65e01b906348 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:53.245148 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd66905-7ac2-4c23-8625-4d8c622dab82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:53.246397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd66905-7ac2-4c23-8625-4d8c622dab82 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:53.247106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd66905-7ac2-4c23-8625-4d8c622dab82 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:53.247441 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd66905-7ac2-4c23-8625-4d8c622dab82 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:53.248380 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd66905-7ac2-4c23-8625-4d8c622dab82 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:53.248682 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd66905-7ac2-4c23-8625-4d8c622dab82 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:53.250264 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd66905-7ac2-4c23-8625-4d8c622dab82 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:53.267956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 391/789] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:53.278753 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd66905-7ac2-4c23-8625-4d8c622dab82 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:53.348441 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:53.355727 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627c14d7-cd9c-495e-aaed-65e01b906348 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:53.411384 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f44a877-d1da-462b-af68-37ea6c67c3c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:53.412367 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f44a877-d1da-462b-af68-37ea6c67c3c6 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:53.412847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f44a877-d1da-462b-af68-37ea6c67c3c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:53.413758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f44a877-d1da-462b-af68-37ea6c67c3c6 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:53.414970 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f44a877-d1da-462b-af68-37ea6c67c3c6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:53.415495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f44a877-d1da-462b-af68-37ea6c67c3c6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:53.417239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f44a877-d1da-462b-af68-37ea6c67c3c6 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:53.428028 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 399/790] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:27:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:53.435377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f44a877-d1da-462b-af68-37ea6c67c3c6 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:53.485159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:53.494581 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd66905-7ac2-4c23-8625-4d8c622dab82 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:53.557035 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73840ea5-e413-4f40-b48c-56cfd78ffad8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:53.557682 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73840ea5-e413-4f40-b48c-56cfd78ffad8 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:53.557994 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73840ea5-e413-4f40-b48c-56cfd78ffad8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:53.558275 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73840ea5-e413-4f40-b48c-56cfd78ffad8 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:53.587489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73840ea5-e413-4f40-b48c-56cfd78ffad8 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:53.588866 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73840ea5-e413-4f40-b48c-56cfd78ffad8 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:53.590740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f44a877-d1da-462b-af68-37ea6c67c3c6 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:53.623918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-73840ea5-e413-4f40-b48c-56cfd78ffad8 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 30 09:27:53.623918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:53.623918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:53.623918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:53.623918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:53.623918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:53.623918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:53.623918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:53.623918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:53.623918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:53.623918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:53.623918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:53.623918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:53.623918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:53.623918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:53.623918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:53.624949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:53.624949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:53.624949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:53.624949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:53.624949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:53.624949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:53.624949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:53.624949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:53.624949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:53.624949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:53.624949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:53.624949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:53.624949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:53.624949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:53.624949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:53.624949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:53.624949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:53.624949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:53.625906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:53.625906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:53.625906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:53.625906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:53.625906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:53.625906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:53.625906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:53.625906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:53.625906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 30 09:27:53.625906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:53.625906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 392/791] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:27:53 2022] GET /identity/v3/users/swift => generated 83 bytes in 347 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:53.630470 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14e0703-146d-4467-87ab-83dc5e4bbd39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:53.631637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14e0703-146d-4467-87ab-83dc5e4bbd39 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:53.632138 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14e0703-146d-4467-87ab-83dc5e4bbd39 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:53.632618 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14e0703-146d-4467-87ab-83dc5e4bbd39 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:53.635756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73840ea5-e413-4f40-b48c-56cfd78ffad8 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:53.644326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14e0703-146d-4467-87ab-83dc5e4bbd39 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:53.646326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14e0703-146d-4467-87ab-83dc5e4bbd39 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:53.663343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-a14e0703-146d-4467-87ab-83dc5e4bbd39 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:53.663343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:53.663343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:53.663343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:53.663343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:53.663343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:53.663343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:53.663343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:53.663343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:53.663343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:53.663343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:53.663343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:53.663343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:53.663343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:53.663343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:53.663343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:53.664339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:53.664339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:53.664339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:53.664339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:53.664339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:53.664339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:53.664339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:53.664339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:53.664339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:53.664339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:53.664339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:53.664339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:53.664339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:53.664339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:53.664339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:53.664339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:53.664339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:53.665273 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:53.665273 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:53.665273 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:53.667982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 400/792] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:53.675567 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14e0703-146d-4467-87ab-83dc5e4bbd39 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:53.698919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:53.832540 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73840ea5-e413-4f40-b48c-56cfd78ffad8 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:53.856573 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14e0703-146d-4467-87ab-83dc5e4bbd39 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:53.896468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0d75bd-5a72-4f8b-bd49-7f3a66d790d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:53.897000 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0d75bd-5a72-4f8b-bd49-7f3a66d790d9 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:53.897281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0d75bd-5a72-4f8b-bd49-7f3a66d790d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:53.897554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0d75bd-5a72-4f8b-bd49-7f3a66d790d9 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:53.898277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0d75bd-5a72-4f8b-bd49-7f3a66d790d9 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:53.898584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0d75bd-5a72-4f8b-bd49-7f3a66d790d9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:53.900823 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0d75bd-5a72-4f8b-bd49-7f3a66d790d9 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:53.911191 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ca080d-11a2-4bfe-8404-2c4b5600842a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:53.911796 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ca080d-11a2-4bfe-8404-2c4b5600842a None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:53.912080 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ca080d-11a2-4bfe-8404-2c4b5600842a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:53.912350 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ca080d-11a2-4bfe-8404-2c4b5600842a None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:53.912987 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ca080d-11a2-4bfe-8404-2c4b5600842a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:53.913294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ca080d-11a2-4bfe-8404-2c4b5600842a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:53.914675 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ca080d-11a2-4bfe-8404-2c4b5600842a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:53.933183 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 393/793] 10.222.0.39 () {56 vars in 1135 bytes} [Fri Sep 30 09:27:53 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:53.938629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0d75bd-5a72-4f8b-bd49-7f3a66d790d9 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:53.939028 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 401/794] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:53.946331 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ca080d-11a2-4bfe-8404-2c4b5600842a None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:53.970251 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:54.090944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:54.104223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0d75bd-5a72-4f8b-bd49-7f3a66d790d9 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:54.135464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ca080d-11a2-4bfe-8404-2c4b5600842a None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:54.153202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4c1264-1189-4108-b371-7d68df84d639 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:54.153775 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4c1264-1189-4108-b371-7d68df84d639 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:54.154053 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4c1264-1189-4108-b371-7d68df84d639 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:54.154324 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4c1264-1189-4108-b371-7d68df84d639 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:54.168193 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4c1264-1189-4108-b371-7d68df84d639 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:54.169656 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4c1264-1189-4108-b371-7d68df84d639 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:54.197094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-6c4c1264-1189-4108-b371-7d68df84d639 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:54.197094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:54.197094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:54.197094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:54.197094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:54.197094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:54.197094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:54.197094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:54.197094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:54.197094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:54.197094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:54.197094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:54.197094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:54.197094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:54.197094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:54.197094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:54.198113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:54.198113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:54.198113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:54.198113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:54.198113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:54.198113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:54.198113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:54.198113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:54.198113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:54.198113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:54.198113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:54.198113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:54.198113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:54.198113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:54.198113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:54.198113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:54.198113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:54.199621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:54.199621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:54.199621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:54.205261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 394/795] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:54.210117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4c1264-1189-4108-b371-7d68df84d639 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:54.211565 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c6987e-91f6-4de0-834f-61670628213d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:54.212201 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c6987e-91f6-4de0-834f-61670628213d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:54.212479 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c6987e-91f6-4de0-834f-61670628213d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:54.212752 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c6987e-91f6-4de0-834f-61670628213d None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:54.234494 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c6987e-91f6-4de0-834f-61670628213d None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:54.235997 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c6987e-91f6-4de0-834f-61670628213d None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:54.264410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-a8c6987e-91f6-4de0-834f-61670628213d None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 30 09:27:54.264410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:54.264410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:54.264410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:54.264410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:54.264410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:54.264410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:54.264410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:54.264410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:54.264410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:54.264410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:54.264410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:54.264410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:54.264410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:54.264410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:54.264410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:54.266739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:54.266739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:54.266739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:54.266739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:54.266739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:54.266739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:54.266739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:54.266739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:54.266739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:54.266739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:54.266739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:54.266739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:54.266739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:54.266739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:54.266739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:54.266739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:54.266739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:54.266739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:54.269069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:54.269069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:54.269069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:54.269069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:54.269069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:54.269069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:54.269069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:54.269069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:54.269069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 30 09:27:54.269069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:54.269069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 402/796] 10.222.0.39 () {56 vars in 1117 bytes} [Fri Sep 30 09:27:53 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:54.274942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c6987e-91f6-4de0-834f-61670628213d None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:54.299032 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:54.336491 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4c1264-1189-4108-b371-7d68df84d639 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:54.374983 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6ea488-ade1-44af-aa53-ef62243386b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:54.378621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6ea488-ade1-44af-aa53-ef62243386b6 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:54.378621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6ea488-ade1-44af-aa53-ef62243386b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:54.378621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6ea488-ade1-44af-aa53-ef62243386b6 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:54.378621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6ea488-ade1-44af-aa53-ef62243386b6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:54.378621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6ea488-ade1-44af-aa53-ef62243386b6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:54.379126 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6ea488-ade1-44af-aa53-ef62243386b6 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:54.393240 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 395/797] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:54.399795 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6ea488-ade1-44af-aa53-ef62243386b6 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:54.496588 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c6987e-91f6-4de0-834f-61670628213d None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:54.564500 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:54.570795 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6ea488-ade1-44af-aa53-ef62243386b6 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:54.595297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c3ccbf-c84e-4c94-91a8-ae840518eaee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:54.596242 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c3ccbf-c84e-4c94-91a8-ae840518eaee None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:54.596710 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c3ccbf-c84e-4c94-91a8-ae840518eaee None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:54.597175 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c3ccbf-c84e-4c94-91a8-ae840518eaee None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:54.598059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c3ccbf-c84e-4c94-91a8-ae840518eaee None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:54.598514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c3ccbf-c84e-4c94-91a8-ae840518eaee None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:54.598974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:54.600128 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c3ccbf-c84e-4c94-91a8-ae840518eaee None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:54.608679 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffff83c-c9ea-4415-85c8-2852849540b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:54.609399 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffff83c-c9ea-4415-85c8-2852849540b1 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:54.609843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffff83c-c9ea-4415-85c8-2852849540b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:54.610123 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffff83c-c9ea-4415-85c8-2852849540b1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:54.618241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffff83c-c9ea-4415-85c8-2852849540b1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:54.624578 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffff83c-c9ea-4415-85c8-2852849540b1 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:54.634174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-0ffff83c-c9ea-4415-85c8-2852849540b1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:54.634174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:54.634174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:54.634174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:54.634174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:54.634174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:54.634174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:54.634174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:54.634174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:54.634174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:54.634174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:54.634174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:54.634174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:54.634174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:54.634174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:54.635145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:54.635145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:54.635145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:54.635145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:54.635145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:54.635145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:54.635145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:54.635145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:54.635145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:54.635145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:54.635145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:54.635145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:54.635145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:54.635145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:54.635145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:54.635145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:54.635145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:54.636078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:54.636078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:54.636078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:54.636078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:54.636078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:54.636078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:54.636078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:54.636078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:54.636078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 396/798] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:54.642615 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 403/799] 10.222.0.39 () {56 vars in 1149 bytes} [Fri Sep 30 09:27:54 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 365 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:54.642781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6894831c-4bfa-4cf4-8405-2c2b81bdeb2e None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:54.642781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6894831c-4bfa-4cf4-8405-2c2b81bdeb2e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:54.642781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6894831c-4bfa-4cf4-8405-2c2b81bdeb2e None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:54.642781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 397/800] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:54.643046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c3ccbf-c84e-4c94-91a8-ae840518eaee None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:54.646829 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950300cb-ea8e-4f96-aea2-cfa7464adaf7 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:54.647125 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950300cb-ea8e-4f96-aea2-cfa7464adaf7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:54.647408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950300cb-ea8e-4f96-aea2-cfa7464adaf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:54.665298 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-950300cb-ea8e-4f96-aea2-cfa7464adaf7 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:54.666497 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-950300cb-ea8e-4f96-aea2-cfa7464adaf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:54.734233 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 398/801] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:54 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 30 09:27:54.735086 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950300cb-ea8e-4f96-aea2-cfa7464adaf7 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:54.759251 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c3ccbf-c84e-4c94-91a8-ae840518eaee None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:54.806188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7759c16f-6d6a-40e9-b814-7901c0eed28b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:54.807324 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7759c16f-6d6a-40e9-b814-7901c0eed28b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:54.807591 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7759c16f-6d6a-40e9-b814-7901c0eed28b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:54.808571 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7759c16f-6d6a-40e9-b814-7901c0eed28b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:54.819279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7759c16f-6d6a-40e9-b814-7901c0eed28b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:54.820805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7759c16f-6d6a-40e9-b814-7901c0eed28b None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:54.836154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-7759c16f-6d6a-40e9-b814-7901c0eed28b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:54.836154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:54.836154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:54.836154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:54.836154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:54.836154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:54.836154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:54.836154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:54.836154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:54.836154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:54.836154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:54.836154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:54.836154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:54.836154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:54.836154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:54.836154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:54.837139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:54.837139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:54.837139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:54.837139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:54.837139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:54.837139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:54.837139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:54.837139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:54.837139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:54.837139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:54.837139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:54.837139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:54.837139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:54.837139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:54.837139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:54.837139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:54.837139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:54.838071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:54.838071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:54.838071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:54.838071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 404/802] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:54.839328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac2e129-9ab0-4881-958e-6d9dc2668f6b None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:54.839673 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac2e129-9ab0-4881-958e-6d9dc2668f6b None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:54.840010 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac2e129-9ab0-4881-958e-6d9dc2668f6b None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:54.845241 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 405/803] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:54.845241 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac2e129-9ab0-4881-958e-6d9dc2668f6b None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:54.855833 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950300cb-ea8e-4f96-aea2-cfa7464adaf7 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:54.875444 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:54.895881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e7f6d4-b127-448a-af6d-9c02772b3913 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:54.897264 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e7f6d4-b127-448a-af6d-9c02772b3913 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:54.897746 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e7f6d4-b127-448a-af6d-9c02772b3913 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:54.898854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:54.899368 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e7f6d4-b127-448a-af6d-9c02772b3913 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:54.901049 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e7f6d4-b127-448a-af6d-9c02772b3913 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:54.901828 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e7f6d4-b127-448a-af6d-9c02772b3913 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:54.903460 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e7f6d4-b127-448a-af6d-9c02772b3913 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:54.918099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 399/804] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:54.919541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00240bf6-5b25-41a8-be13-81934a3c8986 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:54.919988 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00240bf6-5b25-41a8-be13-81934a3c8986 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:54.920463 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00240bf6-5b25-41a8-be13-81934a3c8986 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:54.942372 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-00240bf6-5b25-41a8-be13-81934a3c8986 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:54.943879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-00240bf6-5b25-41a8-be13-81934a3c8986 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:54.981170 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac2e129-9ab0-4881-958e-6d9dc2668f6b None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:55.017171 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c569beb-d880-44ae-b0e3-309e7c0bce58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:55.018115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c569beb-d880-44ae-b0e3-309e7c0bce58 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:55.018633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c569beb-d880-44ae-b0e3-309e7c0bce58 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:55.019104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c569beb-d880-44ae-b0e3-309e7c0bce58 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:55.019910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c569beb-d880-44ae-b0e3-309e7c0bce58 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:55.020393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c569beb-d880-44ae-b0e3-309e7c0bce58 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:55.021994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c569beb-d880-44ae-b0e3-309e7c0bce58 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:55.030623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 400/805] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:54 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:27:55.030623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00240bf6-5b25-41a8-be13-81934a3c8986 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:55.047538 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 406/806] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:55.049072 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c569beb-d880-44ae-b0e3-309e7c0bce58 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:55.152487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00240bf6-5b25-41a8-be13-81934a3c8986 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:55.163405 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:55.172926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:55.175908 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c569beb-d880-44ae-b0e3-309e7c0bce58 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:55.203482 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b2644d-0dbe-44fa-ad63-d5e969d9b2d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:55.204710 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b2644d-0dbe-44fa-ad63-d5e969d9b2d0 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:55.205430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b2644d-0dbe-44fa-ad63-d5e969d9b2d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:55.205902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b2644d-0dbe-44fa-ad63-d5e969d9b2d0 None admin] PATH_INFO: `/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/0e5ad921e6fa41c297696ec7279f0fc4/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:55.208812 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b2644d-0dbe-44fa-ad63-d5e969d9b2d0 None admin] RBAC: Authorizing `identity:create_grant(project_id=c3802ced1fce4997a016d41eca843ef8, user_id=0e5ad921e6fa41c297696ec7279f0fc4, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:55.210454 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b2644d-0dbe-44fa-ad63-d5e969d9b2d0 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:55.212077 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c90c731-b957-4dbf-b7da-f3f73e9f9ac7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:55.212783 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c90c731-b957-4dbf-b7da-f3f73e9f9ac7 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:55.213136 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c90c731-b957-4dbf-b7da-f3f73e9f9ac7 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:55.213485 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c90c731-b957-4dbf-b7da-f3f73e9f9ac7 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:55.214680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c90c731-b957-4dbf-b7da-f3f73e9f9ac7 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:55.216119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c90c731-b957-4dbf-b7da-f3f73e9f9ac7 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:55.226329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-5c90c731-b957-4dbf-b7da-f3f73e9f9ac7 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 30 09:27:55.226329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:55.226329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:27:55.226329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:27:55.226329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:27:55.226329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:27:55.226329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:27:55.226329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:27:55.226329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 30 09:27:55.226329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:55.226329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:27:55.226329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:55.226329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:55.226329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:55.226329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:55.226329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:55.226329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:55.233176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:55.233176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:55.233176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:55.233176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:55.233176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:55.233176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:55.233176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:27:55.233176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:27:55.233176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:27:55.233176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:27:55.233176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:55.233176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:55.233176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:55.233176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:55.233176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:55.233176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:55.234277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:55.234277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:55.234277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:55.234277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:55.234277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:55.234277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:55.234277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:55.234277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:55.234277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:55.234277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:55.234277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:27:55.234277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:27:55.234277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 30 09:27:55.234277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:55.234277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 407/807] 10.222.0.39 () {56 vars in 1129 bytes} [Fri Sep 30 09:27:55 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:55.235320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c90c731-b957-4dbf-b7da-f3f73e9f9ac7 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:55.278860 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 401/808] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:27:55 2022] PUT /identity/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/0e5ad921e6fa41c297696ec7279f0fc4/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:27:55.288731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b2644d-0dbe-44fa-ad63-d5e969d9b2d0 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:55.368199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c90c731-b957-4dbf-b7da-f3f73e9f9ac7 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:55.410248 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5168d7-1162-4221-a413-b84f979fd537 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:55.411343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5168d7-1162-4221-a413-b84f979fd537 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:55.411727 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5168d7-1162-4221-a413-b84f979fd537 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:55.412119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5168d7-1162-4221-a413-b84f979fd537 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:55.426255 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5168d7-1162-4221-a413-b84f979fd537 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:55.427803 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5168d7-1162-4221-a413-b84f979fd537 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:55.439847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b2644d-0dbe-44fa-ad63-d5e969d9b2d0 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:55.442257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-1c5168d7-1162-4221-a413-b84f979fd537 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:55.442257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:55.442257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:55.442257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:55.442257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:55.442257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:55.442257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:55.442257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:55.442257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:55.442257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:55.442257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:55.442257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:55.442257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:55.442257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:55.442257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:55.443414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:55.443414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:55.443414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:55.443414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:55.443414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:55.443414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:55.443414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:55.443414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:55.443414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:55.443414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:55.443414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:55.443414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:55.443414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:55.443414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:55.443414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:55.443414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:55.443414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:55.444645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:55.444645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:55.444645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:55.444645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:55.444645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:55.444645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:55.444645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:55.444645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:55.448346 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 408/809] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:55.449848 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5168d7-1162-4221-a413-b84f979fd537 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:55.478641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:55.490855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b17a131-93bf-4f82-81c2-ad6f98dced80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:55.491486 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b17a131-93bf-4f82-81c2-ad6f98dced80 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:55.511670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b17a131-93bf-4f82-81c2-ad6f98dced80 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:55.511670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b17a131-93bf-4f82-81c2-ad6f98dced80 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:55.511670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b17a131-93bf-4f82-81c2-ad6f98dced80 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:55.511670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b17a131-93bf-4f82-81c2-ad6f98dced80 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:55.511670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b17a131-93bf-4f82-81c2-ad6f98dced80 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:55.511670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 402/810] 10.222.0.39 () {56 vars in 1124 bytes} [Fri Sep 30 09:27:55 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:55.512201 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b17a131-93bf-4f82-81c2-ad6f98dced80 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:55.535688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:55.561975 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5168d7-1162-4221-a413-b84f979fd537 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:55.599719 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5712df6-8d21-437e-b06a-996db6173dae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:55.600355 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5712df6-8d21-437e-b06a-996db6173dae None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:55.600638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5712df6-8d21-437e-b06a-996db6173dae None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:55.600913 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5712df6-8d21-437e-b06a-996db6173dae None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:55.606577 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b17a131-93bf-4f82-81c2-ad6f98dced80 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:55.611393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5712df6-8d21-437e-b06a-996db6173dae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:55.616256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-c5712df6-8d21-437e-b06a-996db6173dae None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:27:55.616256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:55.616256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:55.616256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:55.616256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:55.616256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:55.616256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:55.616256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:55.616256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:55.616256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:55.616256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:55.616256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:55.616256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:55.616256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:55.616256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:55.616256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:55.617232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:55.617232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:55.617232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:55.617232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:55.617232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:55.617232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:55.617232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:55.617232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:55.617232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:55.617232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:55.617232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:55.617232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:55.617232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:55.617232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:55.617232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:55.617232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:55.618111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:55.618111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:55.618111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:55.618111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:55.618111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:55.618111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:55.618111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:55.618111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:55.618111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:55.618111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:55.618111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:27:55.618111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:55.618111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 409/811] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:27:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:55.618858 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e140ce20-f69b-4260-be25-4dd23a2177e1 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:55.619418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e140ce20-f69b-4260-be25-4dd23a2177e1 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:55.619418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e140ce20-f69b-4260-be25-4dd23a2177e1 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:55.621783 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 410/812] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:55.623513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e140ce20-f69b-4260-be25-4dd23a2177e1 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:55.649812 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d29a1f3-94ca-4a67-9f51-7b5035f18a60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:55.650417 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d29a1f3-94ca-4a67-9f51-7b5035f18a60 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:55.650741 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d29a1f3-94ca-4a67-9f51-7b5035f18a60 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:55.651015 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d29a1f3-94ca-4a67-9f51-7b5035f18a60 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:55.652321 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d29a1f3-94ca-4a67-9f51-7b5035f18a60 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:55.652626 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d29a1f3-94ca-4a67-9f51-7b5035f18a60 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:55.654059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d29a1f3-94ca-4a67-9f51-7b5035f18a60 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:55.670459 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 403/813] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:55.670459 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d29a1f3-94ca-4a67-9f51-7b5035f18a60 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:55.718283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:55.736504 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e140ce20-f69b-4260-be25-4dd23a2177e1 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:55.765022 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:55.772444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7d7f0d-9f12-4d5d-9874-726ca5bcb2e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:55.772983 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7d7f0d-9f12-4d5d-9874-726ca5bcb2e5 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:55.773267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7d7f0d-9f12-4d5d-9874-726ca5bcb2e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:55.773541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7d7f0d-9f12-4d5d-9874-726ca5bcb2e5 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:55.774165 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7d7f0d-9f12-4d5d-9874-726ca5bcb2e5 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:55.774471 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7d7f0d-9f12-4d5d-9874-726ca5bcb2e5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:55.774574 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d29a1f3-94ca-4a67-9f51-7b5035f18a60 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:55.775885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7d7f0d-9f12-4d5d-9874-726ca5bcb2e5 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:55.780883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 411/814] 10.222.0.39 () {56 vars in 1124 bytes} [Fri Sep 30 09:27:55 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:55.782114 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec6eaea-0d6f-41e1-a7d1-e67cdd557bad None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:55.782401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec6eaea-0d6f-41e1-a7d1-e67cdd557bad None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:55.782721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec6eaea-0d6f-41e1-a7d1-e67cdd557bad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:55.801302 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-dec6eaea-0d6f-41e1-a7d1-e67cdd557bad None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:55.802436 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-dec6eaea-0d6f-41e1-a7d1-e67cdd557bad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:55.815824 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546b0383-bfdc-4182-af1e-0d1e5aa34a7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:55.816432 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546b0383-bfdc-4182-af1e-0d1e5aa34a7a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:55.816707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546b0383-bfdc-4182-af1e-0d1e5aa34a7a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:55.817077 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546b0383-bfdc-4182-af1e-0d1e5aa34a7a None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:55.817742 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546b0383-bfdc-4182-af1e-0d1e5aa34a7a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:55.818034 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546b0383-bfdc-4182-af1e-0d1e5aa34a7a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:55.819422 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546b0383-bfdc-4182-af1e-0d1e5aa34a7a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:55.832971 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 404/815] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:27:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:55.832971 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546b0383-bfdc-4182-af1e-0d1e5aa34a7a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:55.879478 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 412/816] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:55 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:27:55.880628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec6eaea-0d6f-41e1-a7d1-e67cdd557bad None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:55.950570 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546b0383-bfdc-4182-af1e-0d1e5aa34a7a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:55.975265 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:55.990445 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50db289e-4fe8-442b-bce6-2159d764a769 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:55.991239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50db289e-4fe8-442b-bce6-2159d764a769 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:55.991577 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50db289e-4fe8-442b-bce6-2159d764a769 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:55.991910 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50db289e-4fe8-442b-bce6-2159d764a769 None admin] PATH_INFO: `/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/7d6e0545a40a436d8c63820b795551a9/roles/144dd855f74945548ea65db37acda1c1` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:55.999519 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec6eaea-0d6f-41e1-a7d1-e67cdd557bad None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:56.012729 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50db289e-4fe8-442b-bce6-2159d764a769 None admin] RBAC: Authorizing `identity:create_grant(project_id=c3802ced1fce4997a016d41eca843ef8, user_id=7d6e0545a40a436d8c63820b795551a9, role_id=144dd855f74945548ea65db37acda1c1)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:56.014323 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50db289e-4fe8-442b-bce6-2159d764a769 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:56.036925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8447d065-8395-4ca8-a096-89993006a23f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:56.037761 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8447d065-8395-4ca8-a096-89993006a23f None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:56.038721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8447d065-8395-4ca8-a096-89993006a23f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:56.038721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8447d065-8395-4ca8-a096-89993006a23f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:56.039567 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8447d065-8395-4ca8-a096-89993006a23f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:56.040948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8447d065-8395-4ca8-a096-89993006a23f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:56.095410 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 405/817] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:27:55 2022] PUT /identity/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/7d6e0545a40a436d8c63820b795551a9/roles/144dd855f74945548ea65db37acda1c1 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:27:56.096584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50db289e-4fe8-442b-bce6-2159d764a769 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:56.106371 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:56.127951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 413/818] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:27:55 2022] POST /identity/v3/endpoints => generated 346 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:27:56.188242 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8447d065-8395-4ca8-a096-89993006a23f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:56.295383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50db289e-4fe8-442b-bce6-2159d764a769 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:56.377674 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c745f0d-1aa4-4321-9e67-cf9e591baebb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:56.378244 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c745f0d-1aa4-4321-9e67-cf9e591baebb None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:56.378615 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c745f0d-1aa4-4321-9e67-cf9e591baebb None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:56.378901 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c745f0d-1aa4-4321-9e67-cf9e591baebb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:56.390225 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8447d065-8395-4ca8-a096-89993006a23f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:56.392486 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c745f0d-1aa4-4321-9e67-cf9e591baebb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:56.394061 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c745f0d-1aa4-4321-9e67-cf9e591baebb None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:56.411187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-3c745f0d-1aa4-4321-9e67-cf9e591baebb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:56.411187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:56.411187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:56.411187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:56.411187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:56.411187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:56.411187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:56.411187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:56.411187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:56.411187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:56.411187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:56.411187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:56.411187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:56.411187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:56.411187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:56.411187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:56.412185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:56.412185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:56.412185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:56.412185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:56.412185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:56.412185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:56.412185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:56.412185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:56.412185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:56.412185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:56.412185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:56.412185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:56.412185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:56.412185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:56.412185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:56.412185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:56.412185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:56.413081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:56.413081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:56.413081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:56.413081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 406/819] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 316 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:56.418711 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c745f0d-1aa4-4321-9e67-cf9e591baebb None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:56.433885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ba2880-5b25-4e60-a45d-a9d5fabc4df9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:56.433885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ba2880-5b25-4e60-a45d-a9d5fabc4df9 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:56.433885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ba2880-5b25-4e60-a45d-a9d5fabc4df9 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:56.433885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ba2880-5b25-4e60-a45d-a9d5fabc4df9 None admin] PATH_INFO: `/v3/services/network` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:56.434543 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ba2880-5b25-4e60-a45d-a9d5fabc4df9 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:56.435748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ba2880-5b25-4e60-a45d-a9d5fabc4df9 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:56.440347 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:56.447068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-56ba2880-5b25-4e60-a45d-a9d5fabc4df9 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 30 09:27:56.447068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:56.447068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:27:56.447068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:27:56.447068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:27:56.447068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:27:56.447068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:27:56.447068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:27:56.447068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 30 09:27:56.447068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:56.447068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:27:56.447068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:56.447068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:56.447068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:56.447068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:56.447068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:56.447068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:56.448082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:56.448082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:56.448082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:56.448082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:56.448082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:56.448082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:56.448082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:27:56.448082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:27:56.448082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:27:56.448082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:27:56.448082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:56.448082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:56.448082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:56.448082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:56.448082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:56.448082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:56.449038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:56.449038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:56.449038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:56.449038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:56.449038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:56.449038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:56.449038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:56.449038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:56.449038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:56.449038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:56.449038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:27:56.449038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:27:56.449038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 30 09:27:56.449038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:56.449038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 414/820] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:56 2022] GET /identity/v3/services/network => generated 88 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:56.454430 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ba2880-5b25-4e60-a45d-a9d5fabc4df9 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:56.495775 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:56.551035 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c745f0d-1aa4-4321-9e67-cf9e591baebb None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:56.604572 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ba2880-5b25-4e60-a45d-a9d5fabc4df9 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:56.612718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653aba4c-74af-4446-8ca6-46b7adfb8423 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:56.613349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653aba4c-74af-4446-8ca6-46b7adfb8423 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:56.613635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653aba4c-74af-4446-8ca6-46b7adfb8423 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:56.613914 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653aba4c-74af-4446-8ca6-46b7adfb8423 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:56.614567 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653aba4c-74af-4446-8ca6-46b7adfb8423 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:56.614899 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653aba4c-74af-4446-8ca6-46b7adfb8423 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:56.616393 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653aba4c-74af-4446-8ca6-46b7adfb8423 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:56.644423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 407/821] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:56.651286 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653aba4c-74af-4446-8ca6-46b7adfb8423 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:56.669972 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6233ff39-2ce7-4b16-8380-4ec3a28f7509 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:56.670967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6233ff39-2ce7-4b16-8380-4ec3a28f7509 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:56.671447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6233ff39-2ce7-4b16-8380-4ec3a28f7509 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:56.671938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6233ff39-2ce7-4b16-8380-4ec3a28f7509 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:56.672814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6233ff39-2ce7-4b16-8380-4ec3a28f7509 None admin] RBAC: Adding query filter params (name=network) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:56.673314 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6233ff39-2ce7-4b16-8380-4ec3a28f7509 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:56.675054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6233ff39-2ce7-4b16-8380-4ec3a28f7509 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:56.680384 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 415/822] 10.222.0.39 () {56 vars in 1112 bytes} [Fri Sep 30 09:27:56 2022] GET /identity/v3/services?name=network => generated 124 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:56.685894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6233ff39-2ce7-4b16-8380-4ec3a28f7509 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:56.755714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:56.820941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6233ff39-2ce7-4b16-8380-4ec3a28f7509 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:56.856142 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653aba4c-74af-4446-8ca6-46b7adfb8423 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:56.867697 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:56.892619 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1349177b-0081-4541-943c-ceb6e5d07911 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:56.893585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1349177b-0081-4541-943c-ceb6e5d07911 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:56.894064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1349177b-0081-4541-943c-ceb6e5d07911 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:56.894658 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1349177b-0081-4541-943c-ceb6e5d07911 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:56.902625 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1349177b-0081-4541-943c-ceb6e5d07911 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:56.904334 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1349177b-0081-4541-943c-ceb6e5d07911 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:56.915828 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c335b256-77f7-4ac7-a0d4-103ffa27779b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:56.916443 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c335b256-77f7-4ac7-a0d4-103ffa27779b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:56.916661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-1349177b-0081-4541-943c-ceb6e5d07911 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 30 09:27:56.916661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:56.916661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:56.916661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:56.916661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:56.916661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:56.916661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:56.916661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:56.916661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:56.916661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:56.916661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:56.916661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:56.916661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:56.916661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:56.916661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:56.916661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:56.917635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:56.917635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:56.917635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:56.917635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:56.917635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:56.917635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:56.917635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:56.917635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:56.917635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:56.917635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:56.917635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:56.917635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:56.917635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:56.917635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:56.917635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:56.917635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:56.917635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:56.917635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:56.918642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:56.918642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:56.918642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:56.918642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:56.918642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:56.918642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:56.918642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:56.918642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:56.918642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 30 09:27:56.918642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:56.919206 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c335b256-77f7-4ac7-a0d4-103ffa27779b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:56.919206 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c335b256-77f7-4ac7-a0d4-103ffa27779b None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:56.919206 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c335b256-77f7-4ac7-a0d4-103ffa27779b None admin] RBAC: Adding query filter params (type=network) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:56.919206 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c335b256-77f7-4ac7-a0d4-103ffa27779b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:56.919449 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c335b256-77f7-4ac7-a0d4-103ffa27779b None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:56.921455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 408/823] 10.222.0.39 () {56 vars in 1093 bytes} [Fri Sep 30 09:27:56 2022] GET /identity/v3/users/nova => generated 82 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:56.927109 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1349177b-0081-4541-943c-ceb6e5d07911 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:56.940776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 416/824] 10.222.0.39 () {56 vars in 1112 bytes} [Fri Sep 30 09:27:56 2022] GET /identity/v3/services?type=network => generated 350 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:56.946298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c335b256-77f7-4ac7-a0d4-103ffa27779b None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:57.030500 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1349177b-0081-4541-943c-ceb6e5d07911 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:57.064798 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f3a964-a889-4313-a497-0880dbf33421 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:57.065794 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f3a964-a889-4313-a497-0880dbf33421 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:57.066271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f3a964-a889-4313-a497-0880dbf33421 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:57.066801 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f3a964-a889-4313-a497-0880dbf33421 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:57.067782 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f3a964-a889-4313-a497-0880dbf33421 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:57.068288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f3a964-a889-4313-a497-0880dbf33421 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:57.069975 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f3a964-a889-4313-a497-0880dbf33421 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:57.089763 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 409/825] 10.222.0.39 () {56 vars in 1133 bytes} [Fri Sep 30 09:27:56 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:57.095698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f3a964-a889-4313-a497-0880dbf33421 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:57.124454 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:57.210665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c335b256-77f7-4ac7-a0d4-103ffa27779b None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:57.252299 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f3a964-a889-4313-a497-0880dbf33421 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:57.261056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:57.282483 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5cc503-98e6-4602-ad3a-e1e16825365f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:57.283144 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5cc503-98e6-4602-ad3a-e1e16825365f None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:57.283412 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5cc503-98e6-4602-ad3a-e1e16825365f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:57.283686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5cc503-98e6-4602-ad3a-e1e16825365f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:57.284353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5cc503-98e6-4602-ad3a-e1e16825365f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:57.286909 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5cc503-98e6-4602-ad3a-e1e16825365f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:57.293699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f9ecc8-f038-4e54-a634-0b903504fec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:57.294553 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f9ecc8-f038-4e54-a634-0b903504fec3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:57.295092 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f9ecc8-f038-4e54-a634-0b903504fec3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:57.295560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f9ecc8-f038-4e54-a634-0b903504fec3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:57.375272 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f9ecc8-f038-4e54-a634-0b903504fec3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:57.377084 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f9ecc8-f038-4e54-a634-0b903504fec3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:57.386745 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 417/826] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:27:56 2022] POST /identity/v3/endpoints => generated 337 bytes in 441 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:27:57.415445 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:57.519731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 410/827] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:57 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 30 09:27:57.525750 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5cc503-98e6-4602-ad3a-e1e16825365f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:57.675124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4cc336-0a81-4360-a529-3dcd9ea1f937 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:57.675476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4cc336-0a81-4360-a529-3dcd9ea1f937 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:57.675743 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4cc336-0a81-4360-a529-3dcd9ea1f937 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:57.677406 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 411/828] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:27:57.683938 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fed7fd-f66a-4597-a129-44ac81ca2d67 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:57.684225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fed7fd-f66a-4597-a129-44ac81ca2d67 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:57.684499 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fed7fd-f66a-4597-a129-44ac81ca2d67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:57.689257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5cc503-98e6-4602-ad3a-e1e16825365f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:57.711701 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-71fed7fd-f66a-4597-a129-44ac81ca2d67 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:57.713351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-71fed7fd-f66a-4597-a129-44ac81ca2d67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:57.733996 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5f4833-c58f-4bd8-ba46-d7b3355b0c38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:57.735064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5f4833-c58f-4bd8-ba46-d7b3355b0c38 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:57.735341 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5f4833-c58f-4bd8-ba46-d7b3355b0c38 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:57.735615 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5f4833-c58f-4bd8-ba46-d7b3355b0c38 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:57.750375 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5f4833-c58f-4bd8-ba46-d7b3355b0c38 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:57.752021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5f4833-c58f-4bd8-ba46-d7b3355b0c38 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:57.771866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-dd5f4833-c58f-4bd8-ba46-d7b3355b0c38 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:57.771866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:57.771866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:57.771866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:57.771866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:57.771866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:57.771866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:57.771866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:57.771866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:57.771866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:57.771866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:57.771866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:57.771866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:57.771866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:57.771866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:57.771866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:57.772852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:57.772852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:57.772852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:57.772852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:57.772852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:57.772852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:57.772852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:57.772852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:57.772852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:57.772852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:57.772852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:57.772852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:57.772852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:57.772852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:57.772852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:57.772852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:57.772852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:57.773773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:57.773773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:57.773773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:57.773773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 418/829] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:57.778858 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5f4833-c58f-4bd8-ba46-d7b3355b0c38 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:57.798258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 412/830] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:57 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 30 09:27:57.818718 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:57.893373 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5f4833-c58f-4bd8-ba46-d7b3355b0c38 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:57.929344 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7586eb-98b3-4918-ad19-b4fd53e0e767 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:57.929949 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7586eb-98b3-4918-ad19-b4fd53e0e767 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:57.930230 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7586eb-98b3-4918-ad19-b4fd53e0e767 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:57.930506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7586eb-98b3-4918-ad19-b4fd53e0e767 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:57.931174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7586eb-98b3-4918-ad19-b4fd53e0e767 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:57.931476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7586eb-98b3-4918-ad19-b4fd53e0e767 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:57.933373 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7586eb-98b3-4918-ad19-b4fd53e0e767 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:57.945486 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 419/831] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:57.951179 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fed7fd-f66a-4597-a129-44ac81ca2d67 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:58.014193 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:58.054985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fed7fd-f66a-4597-a129-44ac81ca2d67 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:58.075306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7586eb-98b3-4918-ad19-b4fd53e0e767 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:58.089963 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7fab3c-6335-4bc0-ba78-fe8641444a24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:58.090618 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7fab3c-6335-4bc0-ba78-fe8641444a24 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:58.092455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7fab3c-6335-4bc0-ba78-fe8641444a24 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:58.092455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7fab3c-6335-4bc0-ba78-fe8641444a24 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:58.098657 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7fab3c-6335-4bc0-ba78-fe8641444a24 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:58.100095 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7fab3c-6335-4bc0-ba78-fe8641444a24 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:58.111429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-5f7fab3c-6335-4bc0-ba78-fe8641444a24 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:58.111429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:58.111429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:58.111429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:58.111429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:58.111429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:58.111429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:58.111429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:58.111429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:58.111429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:58.111429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:58.111429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:58.111429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:27:58.111429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:27:58.111429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:27:58.112348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:27:58.112348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:58.112348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:58.112348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:58.112348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:58.112348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:58.112348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:58.112348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:58.112348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:58.112348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:58.112348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:58.112348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:58.112348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:58.112348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:58.112348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:58.112348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:58.112348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:58.113261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:27:58.113261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:27:58.113261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:27:58.113261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:27:58.113261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:27:58.113261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:27:58.113261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:27:58.113261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:58.113261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 413/832] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:27:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:58.119001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7fab3c-6335-4bc0-ba78-fe8641444a24 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:58.196527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:58.209384 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7586eb-98b3-4918-ad19-b4fd53e0e767 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:58.226001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7fab3c-6335-4bc0-ba78-fe8641444a24 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:58.244187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:58.246143 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9b9128-b0bd-4636-b2e8-bb7c594cd27a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:58.246926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9b9128-b0bd-4636-b2e8-bb7c594cd27a None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:58.247302 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9b9128-b0bd-4636-b2e8-bb7c594cd27a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:58.247677 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9b9128-b0bd-4636-b2e8-bb7c594cd27a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:58.258926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9b9128-b0bd-4636-b2e8-bb7c594cd27a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:58.264842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-3c9b9128-b0bd-4636-b2e8-bb7c594cd27a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:27:58.264842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:58.264842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:58.264842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:58.264842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:58.264842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:58.264842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:58.264842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:58.264842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:58.264842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:58.264842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:58.264842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:58.264842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:58.264842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:58.264842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:58.264842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:58.265944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:58.265944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:58.265944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:58.265944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:58.265944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:58.265944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:58.265944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:58.265944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:58.265944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:58.265944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:58.265944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:58.265944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:58.265944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:58.265944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:58.265944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:58.265944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:58.266861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:58.266861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:58.266861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:58.266861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:58.266861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:58.266861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:58.266861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:58.266861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:58.266861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:58.266861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:58.266861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:27:58.266861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:58.267518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 420/833] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:27:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:58.267891 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8180370b-9aae-442b-983c-f4cbc874d7fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:58.268717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8180370b-9aae-442b-983c-f4cbc874d7fe None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:58.269009 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8180370b-9aae-442b-983c-f4cbc874d7fe None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:58.269279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8180370b-9aae-442b-983c-f4cbc874d7fe None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:58.269991 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8180370b-9aae-442b-983c-f4cbc874d7fe None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:58.270296 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8180370b-9aae-442b-983c-f4cbc874d7fe None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:58.275852 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8180370b-9aae-442b-983c-f4cbc874d7fe None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:58.275988 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9b9128-b0bd-4636-b2e8-bb7c594cd27a None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:58.286659 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 414/834] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:27:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:58.292268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8180370b-9aae-442b-983c-f4cbc874d7fe None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:58.404490 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8180370b-9aae-442b-983c-f4cbc874d7fe None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:58.413580 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9b9128-b0bd-4636-b2e8-bb7c594cd27a None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:58.446878 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f3855b-7576-44c6-816d-03116a9bbb74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:58.447879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f3855b-7576-44c6-816d-03116a9bbb74 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:58.448526 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f3855b-7576-44c6-816d-03116a9bbb74 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:58.449323 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f3855b-7576-44c6-816d-03116a9bbb74 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:58.450529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f3855b-7576-44c6-816d-03116a9bbb74 None admin] RBAC: Adding query filter params (role.id=e874c311084c4c9fb7691bce461d319b, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=7ff68fb87700423ab53c90520a6bd3b9) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:58.451086 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f3855b-7576-44c6-816d-03116a9bbb74 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:58.452717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f3855b-7576-44c6-816d-03116a9bbb74 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:58.458418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beddeb53-546e-4f36-80f8-2858b1de22d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:58.459056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beddeb53-546e-4f36-80f8-2858b1de22d4 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:58.459335 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beddeb53-546e-4f36-80f8-2858b1de22d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:58.459605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beddeb53-546e-4f36-80f8-2858b1de22d4 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:58.466627 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beddeb53-546e-4f36-80f8-2858b1de22d4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:58.466627 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beddeb53-546e-4f36-80f8-2858b1de22d4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:58.466627 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beddeb53-546e-4f36-80f8-2858b1de22d4 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:58.466953 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 415/835] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:27:58 2022] GET /identity/v3/role_assignments?user.id=7ff68fb87700423ab53c90520a6bd3b9&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=e874c311084c4c9fb7691bce461d319b => generated 608 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:58.475907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 421/836] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:27:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:58.482268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f3855b-7576-44c6-816d-03116a9bbb74 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:58.544534 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:58.610369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d49324-c50e-43cd-9092-c2ac8a1aa471 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:58.610736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d49324-c50e-43cd-9092-c2ac8a1aa471 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:58.610960 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d49324-c50e-43cd-9092-c2ac8a1aa471 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:58.612004 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f3855b-7576-44c6-816d-03116a9bbb74 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:58.612724 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 422/837] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:27:58.619620 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c81cc71-0194-4499-9b3a-5ba2c209f357 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:58.619923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c81cc71-0194-4499-9b3a-5ba2c209f357 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:58.620730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c81cc71-0194-4499-9b3a-5ba2c209f357 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:58.643525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-6c81cc71-0194-4499-9b3a-5ba2c209f357 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:58.644782 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-6c81cc71-0194-4499-9b3a-5ba2c209f357 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:58.654008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:58.658201 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629e0ba8-c733-4291-94d2-ac3f2a714fa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:58.658876 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629e0ba8-c733-4291-94d2-ac3f2a714fa5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:58.659100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629e0ba8-c733-4291-94d2-ac3f2a714fa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:58.659376 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629e0ba8-c733-4291-94d2-ac3f2a714fa5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:58.673501 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629e0ba8-c733-4291-94d2-ac3f2a714fa5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:58.675667 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629e0ba8-c733-4291-94d2-ac3f2a714fa5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:58.692281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-629e0ba8-c733-4291-94d2-ac3f2a714fa5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:58.692281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:58.692281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:58.692281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:58.692281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:58.692281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:58.692281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:58.692281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:58.692281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:58.692281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:58.692281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:58.692281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:58.692281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:58.692281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:58.692281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:58.692281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:58.693308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:58.693308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:58.693308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:58.693308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:58.693308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:58.693308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:58.693308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:58.693308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:58.693308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:58.693308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:58.693308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:58.693308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:58.693308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:58.693308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:58.693308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:58.693308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:58.693308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:58.694221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:58.694221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:58.694221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:58.694221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 416/838] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:58.699202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629e0ba8-c733-4291-94d2-ac3f2a714fa5 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:58.738714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 423/839] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:58 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 30 09:27:58.801720 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:58.810064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629e0ba8-c733-4291-94d2-ac3f2a714fa5 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:58.847538 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7c3889-0069-48fc-ae22-861ece56bc65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:58.848160 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7c3889-0069-48fc-ae22-861ece56bc65 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:58.848445 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7c3889-0069-48fc-ae22-861ece56bc65 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:58.848730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7c3889-0069-48fc-ae22-861ece56bc65 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:58.849371 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7c3889-0069-48fc-ae22-861ece56bc65 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:58.849682 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7c3889-0069-48fc-ae22-861ece56bc65 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:58.851074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7c3889-0069-48fc-ae22-861ece56bc65 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:58.863841 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 417/840] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:27:58.869539 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c81cc71-0194-4499-9b3a-5ba2c209f357 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:58.907634 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2321b40e-5244-4def-a1cd-a5c40b57bcf1 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:58.907982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2321b40e-5244-4def-a1cd-a5c40b57bcf1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:58.908181 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2321b40e-5244-4def-a1cd-a5c40b57bcf1 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:58.909829 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 418/841] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:27:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:27:58.915982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26eef539-4086-483a-b310-cf0a78e48415 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:58.916268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26eef539-4086-483a-b310-cf0a78e48415 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:58.916543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26eef539-4086-483a-b310-cf0a78e48415 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:58.935955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-26eef539-4086-483a-b310-cf0a78e48415 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:27:58.937150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-26eef539-4086-483a-b310-cf0a78e48415 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:27:58.988697 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c81cc71-0194-4499-9b3a-5ba2c209f357 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:59.013549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 419/842] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:27:58 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:27:59.019448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26eef539-4086-483a-b310-cf0a78e48415 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:59.038134 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32533fb4-501b-421b-9bb6-a885d73e5df5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:59.038804 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32533fb4-501b-421b-9bb6-a885d73e5df5 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:59.039233 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32533fb4-501b-421b-9bb6-a885d73e5df5 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:59.039636 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32533fb4-501b-421b-9bb6-a885d73e5df5 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:59.049239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32533fb4-501b-421b-9bb6-a885d73e5df5 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:59.050743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32533fb4-501b-421b-9bb6-a885d73e5df5 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:59.063598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-32533fb4-501b-421b-9bb6-a885d73e5df5 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 30 09:27:59.063598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:59.063598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:59.063598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:59.063598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:59.063598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:59.063598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:59.063598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:59.063598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:59.063598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:59.063598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:59.063598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:59.063598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:27:59.063598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:27:59.063598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:27:59.063598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:27:59.064689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:59.064689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:59.064689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:27:59.064689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:59.064689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:27:59.064689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:27:59.064689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:59.064689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:59.064689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:59.064689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:59.064689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:59.064689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:59.064689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:59.064689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:59.064689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:59.064689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:59.064689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:59.064689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:59.065662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:59.065662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:59.065662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:27:59.065662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:27:59.065662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:27:59.065662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:27:59.065662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:27:59.065662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:27:59.065662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 30 09:27:59.065662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:27:59.065662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 424/843] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:27:58 2022] GET /identity/v3/users/swift => generated 83 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:59.075734 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32533fb4-501b-421b-9bb6-a885d73e5df5 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:59.082452 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:59.117952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:59.133850 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26eef539-4086-483a-b310-cf0a78e48415 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:59.169776 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48803987-d67e-4fd6-a423-a44c1b3222d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:59.170324 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48803987-d67e-4fd6-a423-a44c1b3222d8 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:59.170616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48803987-d67e-4fd6-a423-a44c1b3222d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:59.170896 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48803987-d67e-4fd6-a423-a44c1b3222d8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:59.181233 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48803987-d67e-4fd6-a423-a44c1b3222d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:27:59.186332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-48803987-d67e-4fd6-a423-a44c1b3222d8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:59.186332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:59.186332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:59.186332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:59.186332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:59.186332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:59.186332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:59.186332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:59.186332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:59.186332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:59.186332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:59.186332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:59.186332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:27:59.186332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:27:59.186332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:27:59.186332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:27:59.187366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:27:59.187366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:27:59.187366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:27:59.187366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:27:59.187366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:59.187366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:59.187366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:59.187366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:59.187366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:59.187366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:59.187366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:59.187366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:59.187366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:59.187366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:59.187366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:59.187366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:59.188245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:59.188245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:59.188245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:59.188245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:59.188245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:27:59.188245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:27:59.188245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:27:59.188245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:27:59.188245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:27:59.188245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:27:59.188245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:27:59.188245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:59.191858 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32533fb4-501b-421b-9bb6-a885d73e5df5 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:59.192325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 420/844] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:27:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:27:59.193143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48803987-d67e-4fd6-a423-a44c1b3222d8 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:59.231480 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cf49c4-bdfc-4c79-83b3-04b7a27f28b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:59.232452 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cf49c4-bdfc-4c79-83b3-04b7a27f28b1 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:59.232940 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cf49c4-bdfc-4c79-83b3-04b7a27f28b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:59.233418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cf49c4-bdfc-4c79-83b3-04b7a27f28b1 None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:59.234421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cf49c4-bdfc-4c79-83b3-04b7a27f28b1 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:59.234971 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cf49c4-bdfc-4c79-83b3-04b7a27f28b1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:59.236731 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cf49c4-bdfc-4c79-83b3-04b7a27f28b1 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:59.256867 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 425/845] 10.222.0.39 () {56 vars in 1135 bytes} [Fri Sep 30 09:27:59 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:59.266648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cf49c4-bdfc-4c79-83b3-04b7a27f28b1 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:59.302100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48803987-d67e-4fd6-a423-a44c1b3222d8 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:59.337660 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df38d8b7-7e43-47ee-a8ea-114bdf15d8ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:59.338269 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df38d8b7-7e43-47ee-a8ea-114bdf15d8ef None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:59.338548 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df38d8b7-7e43-47ee-a8ea-114bdf15d8ef None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:59.339429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df38d8b7-7e43-47ee-a8ea-114bdf15d8ef None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:59.340124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df38d8b7-7e43-47ee-a8ea-114bdf15d8ef None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:59.340427 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df38d8b7-7e43-47ee-a8ea-114bdf15d8ef None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:59.341789 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df38d8b7-7e43-47ee-a8ea-114bdf15d8ef None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:59.344777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:59.350227 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 421/846] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:27:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:59.355431 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df38d8b7-7e43-47ee-a8ea-114bdf15d8ef None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:59.365662 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:59.387443 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cf49c4-bdfc-4c79-83b3-04b7a27f28b1 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:59.422406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00278f4a-06b4-426f-85a3-fc3a1f626175 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:59.423130 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00278f4a-06b4-426f-85a3-fc3a1f626175 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:59.423418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00278f4a-06b4-426f-85a3-fc3a1f626175 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:59.423695 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00278f4a-06b4-426f-85a3-fc3a1f626175 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:59.460805 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df38d8b7-7e43-47ee-a8ea-114bdf15d8ef None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:59.496505 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da2a23d-2cc5-453c-a37e-8d8698b9f569 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:59.497138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da2a23d-2cc5-453c-a37e-8d8698b9f569 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:59.497418 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da2a23d-2cc5-453c-a37e-8d8698b9f569 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:59.497689 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da2a23d-2cc5-453c-a37e-8d8698b9f569 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:59.498263 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da2a23d-2cc5-453c-a37e-8d8698b9f569 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:59.499673 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da2a23d-2cc5-453c-a37e-8d8698b9f569 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:59.502453 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00278f4a-06b4-426f-85a3-fc3a1f626175 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:59.504482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00278f4a-06b4-426f-85a3-fc3a1f626175 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:59.509101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-6da2a23d-2cc5-453c-a37e-8d8698b9f569 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 30 09:27:59.509101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:59.509101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:27:59.509101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:27:59.509101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:27:59.509101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:27:59.509101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:27:59.509101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:27:59.509101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 30 09:27:59.509101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:59.509101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:27:59.509101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:59.509101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:59.509101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:59.509101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:59.509101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:59.509101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:59.510080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:59.510080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:59.510080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:59.510080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:59.510080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:59.510080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:59.510080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:27:59.510080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:27:59.510080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:27:59.510080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:27:59.510080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:59.510080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:59.510080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:59.510080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:59.510080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:59.510080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:59.515958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:59.515958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:59.515958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:59.515958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:59.515958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:59.515958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:59.515958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:59.515958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:59.515958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:59.515958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:59.515958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:27:59.515958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:27:59.515958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 30 09:27:59.515958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:59.515958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 422/847] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:27:59 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:59.516828 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da2a23d-2cc5-453c-a37e-8d8698b9f569 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:59.567368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:59.591816 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:59.613132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 426/848] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:27:59 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 30 09:27:59.613132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00278f4a-06b4-426f-85a3-fc3a1f626175 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:59.626786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da2a23d-2cc5-453c-a37e-8d8698b9f569 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:59.665255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10cc9e1f-3766-4d93-b6dd-f10876dfa866 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:59.665883 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cc9e1f-3766-4d93-b6dd-f10876dfa866 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:59.666161 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cc9e1f-3766-4d93-b6dd-f10876dfa866 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:59.666725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cc9e1f-3766-4d93-b6dd-f10876dfa866 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:59.676925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10cc9e1f-3766-4d93-b6dd-f10876dfa866 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:59.678295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10cc9e1f-3766-4d93-b6dd-f10876dfa866 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:59.692937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-10cc9e1f-3766-4d93-b6dd-f10876dfa866 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:59.692937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:59.692937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:59.692937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:59.692937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:59.692937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:59.692937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:59.692937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:59.692937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:59.692937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:59.692937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:59.692937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:59.692937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:59.692937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:59.692937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:59.692937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:59.693973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:59.693973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:59.693973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:59.693973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:59.693973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:59.693973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:59.693973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:59.693973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:59.693973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:59.693973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:59.693973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:59.693973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:59.693973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:59.693973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:59.693973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:59.693973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:59.693973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:59.694934 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:59.694934 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:59.694934 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:59.694934 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 423/849] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:59.694934 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10cc9e1f-3766-4d93-b6dd-f10876dfa866 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:59.718069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00278f4a-06b4-426f-85a3-fc3a1f626175 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:59.752897 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6ef2b9-1a2f-4fe1-8dc8-2a57feb8f21f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:59.753567 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6ef2b9-1a2f-4fe1-8dc8-2a57feb8f21f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:59.753813 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6ef2b9-1a2f-4fe1-8dc8-2a57feb8f21f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:59.754028 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6ef2b9-1a2f-4fe1-8dc8-2a57feb8f21f None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:59.754673 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6ef2b9-1a2f-4fe1-8dc8-2a57feb8f21f None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:59.754982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6ef2b9-1a2f-4fe1-8dc8-2a57feb8f21f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:59.756595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6ef2b9-1a2f-4fe1-8dc8-2a57feb8f21f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:59.766080 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 427/850] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:27:59 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:59.766080 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6ef2b9-1a2f-4fe1-8dc8-2a57feb8f21f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:59.789032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:59.812280 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10cc9e1f-3766-4d93-b6dd-f10876dfa866 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:59.835628 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:27:59.849174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ec4700-90fb-48de-830c-20e7546112f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:59.849805 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ec4700-90fb-48de-830c-20e7546112f7 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:59.850082 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ec4700-90fb-48de-830c-20e7546112f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:59.850351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ec4700-90fb-48de-830c-20e7546112f7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:59.861261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ec4700-90fb-48de-830c-20e7546112f7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:59.862718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ec4700-90fb-48de-830c-20e7546112f7 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:59.871430 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6ef2b9-1a2f-4fe1-8dc8-2a57feb8f21f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:27:59.878076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-93ec4700-90fb-48de-830c-20e7546112f7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:59.878076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:27:59.878076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:27:59.878076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:27:59.878076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:27:59.878076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:27:59.878076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:27:59.878076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:27:59.878076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:27:59.878076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:27:59.878076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:27:59.878076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:27:59.878076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:27:59.878076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:27:59.878076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:27:59.878076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:27:59.879162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:27:59.879162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:27:59.879162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:27:59.879162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:27:59.879162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:27:59.879162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:27:59.879162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:27:59.879162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:27:59.879162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:27:59.879162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:27:59.879162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:27:59.879162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:27:59.879162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:27:59.879162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:27:59.879162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:27:59.879162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:27:59.879162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:27:59.885325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:27:59.885325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:27:59.885325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:27:59.885325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 424/851] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:27:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:27:59.885325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ec4700-90fb-48de-830c-20e7546112f7 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:59.919664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbda5c7-dd07-4281-b9a9-1c758a02a512 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:27:59.920273 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbda5c7-dd07-4281-b9a9-1c758a02a512 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:27:59.920550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbda5c7-dd07-4281-b9a9-1c758a02a512 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:27:59.920819 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbda5c7-dd07-4281-b9a9-1c758a02a512 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:27:59.921447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbda5c7-dd07-4281-b9a9-1c758a02a512 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:27:59.921745 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbda5c7-dd07-4281-b9a9-1c758a02a512 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:27:59.923611 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbda5c7-dd07-4281-b9a9-1c758a02a512 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:27:59.937528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 428/852] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:27:59.938255 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbda5c7-dd07-4281-b9a9-1c758a02a512 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:27:59.989723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ec4700-90fb-48de-830c-20e7546112f7 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:00.023984 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:00.028175 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00dedf6-534b-499a-b277-6f4e2a7a0a2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:00.028875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00dedf6-534b-499a-b277-6f4e2a7a0a2b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:00.029159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00dedf6-534b-499a-b277-6f4e2a7a0a2b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:00.029433 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00dedf6-534b-499a-b277-6f4e2a7a0a2b None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:00.030049 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00dedf6-534b-499a-b277-6f4e2a7a0a2b None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:00.030350 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00dedf6-534b-499a-b277-6f4e2a7a0a2b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:00.031814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00dedf6-534b-499a-b277-6f4e2a7a0a2b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:00.036351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 425/853] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:27:59 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:00.037144 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00dedf6-534b-499a-b277-6f4e2a7a0a2b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:00.052982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbda5c7-dd07-4281-b9a9-1c758a02a512 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:00.097227 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:00.098843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97616953-4223-426e-9920-c7c8d8fbbd33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:00.100981 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97616953-4223-426e-9920-c7c8d8fbbd33 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:00.101912 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97616953-4223-426e-9920-c7c8d8fbbd33 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:00.102719 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97616953-4223-426e-9920-c7c8d8fbbd33 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:00.103611 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97616953-4223-426e-9920-c7c8d8fbbd33 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:00.104487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97616953-4223-426e-9920-c7c8d8fbbd33 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:00.107151 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97616953-4223-426e-9920-c7c8d8fbbd33 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:00.124975 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 429/854] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:27:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:00.130260 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97616953-4223-426e-9920-c7c8d8fbbd33 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:00.176339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00dedf6-534b-499a-b277-6f4e2a7a0a2b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:00.226031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0640d437-7d45-4b4e-805d-823cddfd8108 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:00.226456 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0640d437-7d45-4b4e-805d-823cddfd8108 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:00.226872 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0640d437-7d45-4b4e-805d-823cddfd8108 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:00.227151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0640d437-7d45-4b4e-805d-823cddfd8108 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:00.236535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0640d437-7d45-4b4e-805d-823cddfd8108 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:00.238224 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0640d437-7d45-4b4e-805d-823cddfd8108 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:00.254522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-0640d437-7d45-4b4e-805d-823cddfd8108 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 30 09:28:00.254522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:00.254522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:00.254522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:00.254522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:00.254522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:00.254522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:00.254522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:00.254522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:00.254522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:00.254522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:00.254522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:00.254522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:28:00.254522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:28:00.254522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:28:00.254522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:28:00.255566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:00.255566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:00.255566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:28:00.255566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:28:00.255566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:28:00.255566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:28:00.255566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:00.255566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:00.255566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:00.255566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:00.255566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:00.255566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:00.255566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:00.255566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:00.255566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:00.255566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:00.255566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:00.255566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:00.256781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:00.256781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:00.256781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:28:00.256781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:28:00.256781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:28:00.256781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:28:00.256781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:28:00.256781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:28:00.256781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 30 09:28:00.256781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:00.258756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 426/855] 10.222.0.39 () {56 vars in 1117 bytes} [Fri Sep 30 09:28:00 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:00.262400 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0640d437-7d45-4b4e-805d-823cddfd8108 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:00.281759 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97616953-4223-426e-9920-c7c8d8fbbd33 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:00.319665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a068f75c-e9f8-4d5e-aa58-4b257606b6b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:00.320298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a068f75c-e9f8-4d5e-aa58-4b257606b6b0 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:00.320573 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a068f75c-e9f8-4d5e-aa58-4b257606b6b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:00.320841 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a068f75c-e9f8-4d5e-aa58-4b257606b6b0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:00.328815 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a068f75c-e9f8-4d5e-aa58-4b257606b6b0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:00.330482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a068f75c-e9f8-4d5e-aa58-4b257606b6b0 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:00.342818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-a068f75c-e9f8-4d5e-aa58-4b257606b6b0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:28:00.342818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:00.342818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:00.342818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:00.342818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:00.342818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:00.342818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:00.342818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:00.342818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:00.342818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:00.342818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:00.342818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:00.342818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:28:00.342818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:28:00.342818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:28:00.343777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:28:00.343777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:00.343777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:00.343777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:00.343777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:00.343777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:00.343777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:00.343777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:00.343777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:00.343777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:00.343777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:00.343777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:00.343777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:00.343777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:00.343777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:00.343777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:00.343777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:00.345098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:28:00.345098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:28:00.345098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:28:00.345098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:28:00.345098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:28:00.345098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:28:00.345098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:28:00.345098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:00.345098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 430/856] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:28:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:00.349575 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a068f75c-e9f8-4d5e-aa58-4b257606b6b0 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:00.355757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:00.374769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:00.379165 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0640d437-7d45-4b4e-805d-823cddfd8108 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:00.417963 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648b013c-5c57-4c14-95d9-3bf517d34021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:00.419073 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648b013c-5c57-4c14-95d9-3bf517d34021 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:00.419604 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648b013c-5c57-4c14-95d9-3bf517d34021 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:00.420102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648b013c-5c57-4c14-95d9-3bf517d34021 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:00.421026 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648b013c-5c57-4c14-95d9-3bf517d34021 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:00.421548 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648b013c-5c57-4c14-95d9-3bf517d34021 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:00.423436 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648b013c-5c57-4c14-95d9-3bf517d34021 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:00.450484 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 427/857] 10.222.0.39 () {56 vars in 1149 bytes} [Fri Sep 30 09:28:00 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:00.452306 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648b013c-5c57-4c14-95d9-3bf517d34021 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:00.460506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a068f75c-e9f8-4d5e-aa58-4b257606b6b0 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:00.496433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b85612-ca4e-4d58-ab80-d0b503020075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:00.497044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b85612-ca4e-4d58-ab80-d0b503020075 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:00.497324 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b85612-ca4e-4d58-ab80-d0b503020075 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:00.497593 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b85612-ca4e-4d58-ab80-d0b503020075 None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:00.498390 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b85612-ca4e-4d58-ab80-d0b503020075 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:00.498732 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b85612-ca4e-4d58-ab80-d0b503020075 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:00.500279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b85612-ca4e-4d58-ab80-d0b503020075 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:00.519116 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 431/858] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:28:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:00.520005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b85612-ca4e-4d58-ab80-d0b503020075 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:00.561071 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648b013c-5c57-4c14-95d9-3bf517d34021 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:00.589381 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:00.603494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c109d2c-e0e2-4aec-b3e2-036a355a5c38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:00.605012 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c109d2c-e0e2-4aec-b3e2-036a355a5c38 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:00.605528 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c109d2c-e0e2-4aec-b3e2-036a355a5c38 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:00.606056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c109d2c-e0e2-4aec-b3e2-036a355a5c38 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:00.633115 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:00.638733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b85612-ca4e-4d58-ab80-d0b503020075 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:00.676862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b4f16c-2003-44c6-bc99-860bbb691f70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:00.677905 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b4f16c-2003-44c6-bc99-860bbb691f70 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:00.678369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b4f16c-2003-44c6-bc99-860bbb691f70 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:00.678871 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b4f16c-2003-44c6-bc99-860bbb691f70 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:00.679868 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b4f16c-2003-44c6-bc99-860bbb691f70 None admin] RBAC: Adding query filter params (role.id=e874c311084c4c9fb7691bce461d319b, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=0e5ad921e6fa41c297696ec7279f0fc4) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:00.680400 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b4f16c-2003-44c6-bc99-860bbb691f70 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:00.682190 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b4f16c-2003-44c6-bc99-860bbb691f70 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:00.685849 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c109d2c-e0e2-4aec-b3e2-036a355a5c38 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:00.687879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c109d2c-e0e2-4aec-b3e2-036a355a5c38 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:00.690722 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 432/859] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:28:00 2022] GET /identity/v3/role_assignments?user.id=0e5ad921e6fa41c297696ec7279f0fc4&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=e874c311084c4c9fb7691bce461d319b => generated 608 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:00.790292 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98073ade-5852-4bff-bf69-efa1b859f300 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:00.790688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98073ade-5852-4bff-bf69-efa1b859f300 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:00.790862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98073ade-5852-4bff-bf69-efa1b859f300 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:00.792907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 433/860] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:28:00.800077 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45aa04ba-6bdd-4f50-a81b-d798c7c0fd5e None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:00.800380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45aa04ba-6bdd-4f50-a81b-d798c7c0fd5e None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:00.800631 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45aa04ba-6bdd-4f50-a81b-d798c7c0fd5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:00.801362 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 428/861] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:28:00 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 30 09:28:00.807056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c109d2c-e0e2-4aec-b3e2-036a355a5c38 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:00.820441 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-45aa04ba-6bdd-4f50-a81b-d798c7c0fd5e None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:00.821680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-45aa04ba-6bdd-4f50-a81b-d798c7c0fd5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:00.876952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:00.900720 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 434/862] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:00 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:00.926765 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c109d2c-e0e2-4aec-b3e2-036a355a5c38 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:00.961320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f26ba80-fb0b-4f98-b373-7e4c4f7a847f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:00.961953 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f26ba80-fb0b-4f98-b373-7e4c4f7a847f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:00.962231 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f26ba80-fb0b-4f98-b373-7e4c4f7a847f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:00.962503 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f26ba80-fb0b-4f98-b373-7e4c4f7a847f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:00.973927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f26ba80-fb0b-4f98-b373-7e4c4f7a847f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:00.975399 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f26ba80-fb0b-4f98-b373-7e4c4f7a847f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:00.990623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-7f26ba80-fb0b-4f98-b373-7e4c4f7a847f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:00.990623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:00.990623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:00.990623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:00.990623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:00.990623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:00.990623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:00.990623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:00.990623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:00.990623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:00.990623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:00.990623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:00.990623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:28:00.990623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:28:00.990623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:28:00.990623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:28:00.991603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:00.991603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:00.991603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:00.991603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:00.991603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:00.991603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:00.991603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:00.991603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:00.991603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:00.991603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:00.991603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:00.991603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:00.991603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:00.991603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:00.991603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:00.991603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:00.991603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:28:00.992510 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:28:00.992510 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:00.992510 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:00.992510 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 429/863] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:00.997768 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45aa04ba-6bdd-4f50-a81b-d798c7c0fd5e None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:01.062214 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:01.106998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45aa04ba-6bdd-4f50-a81b-d798c7c0fd5e None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:01.142490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470c1f83-4a21-4472-94bc-aa26fc399270 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:01.143385 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470c1f83-4a21-4472-94bc-aa26fc399270 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:01.143815 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470c1f83-4a21-4472-94bc-aa26fc399270 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:01.144277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470c1f83-4a21-4472-94bc-aa26fc399270 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:01.145053 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470c1f83-4a21-4472-94bc-aa26fc399270 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:01.145515 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470c1f83-4a21-4472-94bc-aa26fc399270 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:01.147063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470c1f83-4a21-4472-94bc-aa26fc399270 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:01.159719 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 435/864] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:28:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:01.167459 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f26ba80-fb0b-4f98-b373-7e4c4f7a847f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:01.168750 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470c1f83-4a21-4472-94bc-aa26fc399270 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:01.296626 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470c1f83-4a21-4472-94bc-aa26fc399270 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:01.310211 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f26ba80-fb0b-4f98-b373-7e4c4f7a847f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:01.319867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:01.320560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:01.344345 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c3a5f5-11b0-4525-8d12-248ea1a8fe13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:01.345423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c3a5f5-11b0-4525-8d12-248ea1a8fe13 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:01.345855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c3a5f5-11b0-4525-8d12-248ea1a8fe13 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:01.346277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c3a5f5-11b0-4525-8d12-248ea1a8fe13 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:01.347104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c3a5f5-11b0-4525-8d12-248ea1a8fe13 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:01.347220 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba1618e-e613-48f8-8cf4-f55e3d99b2da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:01.347873 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba1618e-e613-48f8-8cf4-f55e3d99b2da None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:01.348215 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba1618e-e613-48f8-8cf4-f55e3d99b2da None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:01.348556 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba1618e-e613-48f8-8cf4-f55e3d99b2da None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:01.349454 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c3a5f5-11b0-4525-8d12-248ea1a8fe13 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:01.355652 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba1618e-e613-48f8-8cf4-f55e3d99b2da None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:01.357151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba1618e-e613-48f8-8cf4-f55e3d99b2da None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:01.360817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-d0c3a5f5-11b0-4525-8d12-248ea1a8fe13 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 30 09:28:01.360817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:01.360817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:28:01.360817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:28:01.360817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:28:01.360817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:28:01.360817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:28:01.360817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:01.360817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 30 09:28:01.360817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:01.360817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:28:01.360817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:01.360817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:01.360817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:01.360817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:01.360817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:01.361948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:01.361948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:01.361948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:01.361948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:01.361948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:01.361948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:01.361948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:01.361948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:28:01.361948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:28:01.361948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:28:01.361948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:28:01.361948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:01.361948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:01.361948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:01.361948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:01.361948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:01.361948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:01.363125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:01.363125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:01.363125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:01.363125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:01.363125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:01.363125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:01.363125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:01.363125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:01.363125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:01.363125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:01.363125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:28:01.363125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:01.363125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 30 09:28:01.363125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:01.366446 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 436/865] 10.222.0.39 () {56 vars in 1132 bytes} [Fri Sep 30 09:28:01 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:01.371136 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c3a5f5-11b0-4525-8d12-248ea1a8fe13 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:01.375439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-3ba1618e-e613-48f8-8cf4-f55e3d99b2da None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:28:01.375439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:01.375439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:01.375439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:01.375439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:01.375439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:01.375439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:01.375439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:01.375439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:01.375439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:01.375439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:01.375439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:01.375439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:28:01.375439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:28:01.375439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:28:01.376579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:28:01.376579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:01.376579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:01.376579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:01.376579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:01.376579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:01.376579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:01.376579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:01.376579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:01.376579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:01.376579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:01.376579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:01.376579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:01.376579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:01.376579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:01.376579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:01.376579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:01.377715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:28:01.377715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:28:01.377715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:28:01.377715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:28:01.377715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:28:01.377715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:28:01.377715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:28:01.377715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:01.382841 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 430/866] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:28:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:01.384535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba1618e-e613-48f8-8cf4-f55e3d99b2da None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:01.485070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c3a5f5-11b0-4525-8d12-248ea1a8fe13 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:01.520446 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba1618e-e613-48f8-8cf4-f55e3d99b2da None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:01.527191 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3c125b-2e45-4edc-a2f4-778fa0478278 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:01.527994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3c125b-2e45-4edc-a2f4-778fa0478278 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:01.528429 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3c125b-2e45-4edc-a2f4-778fa0478278 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:01.529384 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3c125b-2e45-4edc-a2f4-778fa0478278 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:01.530166 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3c125b-2e45-4edc-a2f4-778fa0478278 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:01.531109 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3c125b-2e45-4edc-a2f4-778fa0478278 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:01.533216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3c125b-2e45-4edc-a2f4-778fa0478278 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:01.540238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 437/867] 10.222.0.39 () {56 vars in 1126 bytes} [Fri Sep 30 09:28:01 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:01.545760 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3c125b-2e45-4edc-a2f4-778fa0478278 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:01.564510 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21e4198-fb0c-4bf0-9d66-61f3352d170e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:01.565965 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21e4198-fb0c-4bf0-9d66-61f3352d170e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:01.566397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21e4198-fb0c-4bf0-9d66-61f3352d170e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:01.566850 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21e4198-fb0c-4bf0-9d66-61f3352d170e None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:01.567681 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21e4198-fb0c-4bf0-9d66-61f3352d170e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:01.568568 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21e4198-fb0c-4bf0-9d66-61f3352d170e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:01.570760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21e4198-fb0c-4bf0-9d66-61f3352d170e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:01.587434 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 431/868] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:28:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:01.593978 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21e4198-fb0c-4bf0-9d66-61f3352d170e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:01.596845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:01.623724 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:01.661483 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3c125b-2e45-4edc-a2f4-778fa0478278 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:01.697685 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96e32d7-e7d8-4f63-95a5-c6c38f51b761 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:01.699148 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96e32d7-e7d8-4f63-95a5-c6c38f51b761 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:01.699582 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96e32d7-e7d8-4f63-95a5-c6c38f51b761 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:01.700005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96e32d7-e7d8-4f63-95a5-c6c38f51b761 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:01.700831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96e32d7-e7d8-4f63-95a5-c6c38f51b761 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:01.701297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96e32d7-e7d8-4f63-95a5-c6c38f51b761 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:01.702890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96e32d7-e7d8-4f63-95a5-c6c38f51b761 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:01.706521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21e4198-fb0c-4bf0-9d66-61f3352d170e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:01.708565 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 438/869] 10.222.0.39 () {56 vars in 1126 bytes} [Fri Sep 30 09:28:01 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:01.750810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8383bb1-4f5c-4028-ab39-88b31605d316 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:01.751733 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8383bb1-4f5c-4028-ab39-88b31605d316 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:01.752163 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8383bb1-4f5c-4028-ab39-88b31605d316 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:01.752585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8383bb1-4f5c-4028-ab39-88b31605d316 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:01.753475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8383bb1-4f5c-4028-ab39-88b31605d316 None admin] RBAC: Adding query filter params (role.id=144dd855f74945548ea65db37acda1c1, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=7d6e0545a40a436d8c63820b795551a9) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:01.753952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8383bb1-4f5c-4028-ab39-88b31605d316 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:01.755600 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8383bb1-4f5c-4028-ab39-88b31605d316 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:01.763586 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 432/870] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:28:01 2022] GET /identity/v3/role_assignments?user.id=7d6e0545a40a436d8c63820b795551a9&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=144dd855f74945548ea65db37acda1c1 => generated 608 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:02.325902 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8acae0-bcb4-4200-b4f9-0c07f0533478 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:02.326434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8acae0-bcb4-4200-b4f9-0c07f0533478 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:02.326434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8acae0-bcb4-4200-b4f9-0c07f0533478 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:02.327948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 439/871] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:02.335345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b23e131-e204-4d1c-b056-d1bb7933c854 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:02.335858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b23e131-e204-4d1c-b056-d1bb7933c854 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:02.336341 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b23e131-e204-4d1c-b056-d1bb7933c854 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:02.355376 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-2b23e131-e204-4d1c-b056-d1bb7933c854 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:02.356888 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2b23e131-e204-4d1c-b056-d1bb7933c854 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:02.425115 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 433/872] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:02 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:02.695483 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8acae0-bcb4-4200-b4f9-0c07f0533478 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:02.803862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8acae0-bcb4-4200-b4f9-0c07f0533478 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:02.816712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:02.848724 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3863820-4e49-4048-824d-ecd91efafdcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:02.849332 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3863820-4e49-4048-824d-ecd91efafdcf None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:02.849614 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3863820-4e49-4048-824d-ecd91efafdcf None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:02.849883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3863820-4e49-4048-824d-ecd91efafdcf None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:02.850533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3863820-4e49-4048-824d-ecd91efafdcf None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:02.851925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3863820-4e49-4048-824d-ecd91efafdcf None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:02.929469 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 440/873] 10.222.0.39 () {60 vars in 1141 bytes} [Fri Sep 30 09:28:02 2022] POST /identity/v3/services => generated 251 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:28:03.030825 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66164dc6-b486-4419-9c68-047c602c73e7 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:03.031363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66164dc6-b486-4419-9c68-047c602c73e7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:03.031675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66164dc6-b486-4419-9c68-047c602c73e7 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:03.033446 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 434/874] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:03.039660 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbacc76-e6b9-46c9-bb1d-4c611529fda2 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:03.040084 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbacc76-e6b9-46c9-bb1d-4c611529fda2 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:03.040460 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbacc76-e6b9-46c9-bb1d-4c611529fda2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:03.061791 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-fcbacc76-e6b9-46c9-bb1d-4c611529fda2 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:03.063053 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-fcbacc76-e6b9-46c9-bb1d-4c611529fda2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:03.147797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 441/875] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:03 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:03.412902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66164dc6-b486-4419-9c68-047c602c73e7 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:03.438848 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:03.517541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66164dc6-b486-4419-9c68-047c602c73e7 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:03.551507 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925279b6-a86b-4c1a-a778-1ead85ecc89a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:03.552143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925279b6-a86b-4c1a-a778-1ead85ecc89a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:03.552421 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925279b6-a86b-4c1a-a778-1ead85ecc89a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:03.552692 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925279b6-a86b-4c1a-a778-1ead85ecc89a None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:03.553269 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925279b6-a86b-4c1a-a778-1ead85ecc89a None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:03.554634 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925279b6-a86b-4c1a-a778-1ead85ecc89a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:03.564254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-925279b6-a86b-4c1a-a778-1ead85ecc89a None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 30 09:28:03.564254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:03.564254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:28:03.564254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:28:03.564254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:28:03.564254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:28:03.564254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:28:03.564254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:03.564254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 30 09:28:03.564254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:03.564254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:28:03.564254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:03.564254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:03.564254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:03.564254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:03.564254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:03.564254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:03.565277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:03.565277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:03.565277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:03.565277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:03.565277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:03.565277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:03.565277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:28:03.565277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:28:03.565277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:28:03.565277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:28:03.565277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:03.565277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:03.565277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:03.565277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:03.565277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:03.565277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:03.566137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:03.566137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:03.566137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:03.566137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:03.566137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:03.566137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:03.566137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:03.566137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:03.566137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:03.566137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:03.566137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:28:03.566137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:03.566137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 30 09:28:03.566137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:03.566137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 435/876] 10.222.0.39 () {56 vars in 1126 bytes} [Fri Sep 30 09:28:03 2022] GET /identity/v3/services/object-store => generated 93 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:03.571426 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbacc76-e6b9-46c9-bb1d-4c611529fda2 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:03.670634 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:03.676996 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbacc76-e6b9-46c9-bb1d-4c611529fda2 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:03.713511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2cf148-fc8d-43de-8fb4-c622c7e2864c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:03.714123 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2cf148-fc8d-43de-8fb4-c622c7e2864c None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:03.714400 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2cf148-fc8d-43de-8fb4-c622c7e2864c None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:03.714694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2cf148-fc8d-43de-8fb4-c622c7e2864c None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:03.715321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2cf148-fc8d-43de-8fb4-c622c7e2864c None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:03.715682 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2cf148-fc8d-43de-8fb4-c622c7e2864c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:03.717175 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2cf148-fc8d-43de-8fb4-c622c7e2864c None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:03.722712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 442/877] 10.222.0.39 () {56 vars in 1122 bytes} [Fri Sep 30 09:28:03 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:03.731651 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925279b6-a86b-4c1a-a778-1ead85ecc89a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:03.820007 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:03.835239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925279b6-a86b-4c1a-a778-1ead85ecc89a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:03.870780 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73c8fe4-543c-41b1-8d4a-0c944592bbba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:03.872090 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73c8fe4-543c-41b1-8d4a-0c944592bbba None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:03.872544 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73c8fe4-543c-41b1-8d4a-0c944592bbba None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:03.873014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73c8fe4-543c-41b1-8d4a-0c944592bbba None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:03.873819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73c8fe4-543c-41b1-8d4a-0c944592bbba None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:03.874313 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73c8fe4-543c-41b1-8d4a-0c944592bbba None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:03.875920 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73c8fe4-543c-41b1-8d4a-0c944592bbba None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:03.882261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 436/878] 10.222.0.39 () {56 vars in 1122 bytes} [Fri Sep 30 09:28:03 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:04.018259 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a561d494-d893-451e-8598-09f3e2b6f0d2 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:04.018542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a561d494-d893-451e-8598-09f3e2b6f0d2 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:04.018857 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a561d494-d893-451e-8598-09f3e2b6f0d2 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:04.020680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 443/879] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:04.026880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a29e418-9a0b-49da-9afe-aa3b7248c2c0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:04.027169 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a29e418-9a0b-49da-9afe-aa3b7248c2c0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:04.027450 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a29e418-9a0b-49da-9afe-aa3b7248c2c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:04.052251 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4d3e3d-6ee7-4be8-99b7-1ba56f39a39c None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:04.063975 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7a29e418-9a0b-49da-9afe-aa3b7248c2c0 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:04.064424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4d3e3d-6ee7-4be8-99b7-1ba56f39a39c None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:04.064424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4d3e3d-6ee7-4be8-99b7-1ba56f39a39c None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:04.064560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7a29e418-9a0b-49da-9afe-aa3b7248c2c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:04.064632 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 444/880] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:04.064632 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9817138-baf2-4a9e-908d-5654825c550b None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:04.064632 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9817138-baf2-4a9e-908d-5654825c550b None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:04.064632 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9817138-baf2-4a9e-908d-5654825c550b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:04.084830 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-c9817138-baf2-4a9e-908d-5654825c550b None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:04.086225 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c9817138-baf2-4a9e-908d-5654825c550b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:04.134317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 437/881] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:04 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:04.165969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 445/882] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:04 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:04.404477 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a29e418-9a0b-49da-9afe-aa3b7248c2c0 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:04.451629 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9817138-baf2-4a9e-908d-5654825c550b None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:04.489327 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:04.511340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a29e418-9a0b-49da-9afe-aa3b7248c2c0 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:04.548792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ee1ee3-cd21-4a75-b0e4-77523722d57a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:04.549847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ee1ee3-cd21-4a75-b0e4-77523722d57a None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:04.550334 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ee1ee3-cd21-4a75-b0e4-77523722d57a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:04.550886 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ee1ee3-cd21-4a75-b0e4-77523722d57a None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:04.551782 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ee1ee3-cd21-4a75-b0e4-77523722d57a None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:04.553366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ee1ee3-cd21-4a75-b0e4-77523722d57a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:04.567197 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:04.578037 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9817138-baf2-4a9e-908d-5654825c550b None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:04.614174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9973bc25-7a06-4f5f-84f6-2fc247440134 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:04.614848 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9973bc25-7a06-4f5f-84f6-2fc247440134 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:04.615131 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9973bc25-7a06-4f5f-84f6-2fc247440134 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:04.615406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9973bc25-7a06-4f5f-84f6-2fc247440134 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:04.625865 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9973bc25-7a06-4f5f-84f6-2fc247440134 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:28:04.631110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-9973bc25-7a06-4f5f-84f6-2fc247440134 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 30 09:28:04.631110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:04.631110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:04.631110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:04.631110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:04.631110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:04.631110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:04.631110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:04.631110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:04.631110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:04.631110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:04.631110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:04.631110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:28:04.631110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:28:04.631110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:28:04.631110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:28:04.632137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:28:04.632137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:28:04.632137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:28:04.632137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:28:04.632137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:04.632137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:04.632137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:04.632137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:04.632137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:04.632137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:04.632137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:04.632137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:04.632137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:04.632137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:04.632137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:04.632137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:04.633032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:04.633032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:04.633032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:04.633032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:04.633032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:28:04.633032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:28:04.633032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:28:04.633032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:28:04.633032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:28:04.633032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:28:04.633032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 30 09:28:04.633032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:04.633032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 446/883] 10.222.0.39 () {56 vars in 1120 bytes} [Fri Sep 30 09:28:04 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:04.638560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9973bc25-7a06-4f5f-84f6-2fc247440134 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:04.659801 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 438/884] 10.222.0.39 () {60 vars in 1141 bytes} [Fri Sep 30 09:28:04 2022] POST /identity/v3/services => generated 269 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:28:04.767553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9973bc25-7a06-4f5f-84f6-2fc247440134 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:04.814023 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3514214e-4ef1-4444-898e-dbc614750970 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:04.815086 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3514214e-4ef1-4444-898e-dbc614750970 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:04.815613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3514214e-4ef1-4444-898e-dbc614750970 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:04.816093 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3514214e-4ef1-4444-898e-dbc614750970 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:04.817121 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3514214e-4ef1-4444-898e-dbc614750970 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:04.817664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3514214e-4ef1-4444-898e-dbc614750970 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:04.819380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3514214e-4ef1-4444-898e-dbc614750970 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:04.830967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 447/885] 10.222.0.39 () {56 vars in 1115 bytes} [Fri Sep 30 09:28:04 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:04.836597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ee1ee3-cd21-4a75-b0e4-77523722d57a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:04.941476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ee1ee3-cd21-4a75-b0e4-77523722d57a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:04.975641 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ec0c56-20a5-4d87-83f6-1142d5ba8774 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:04.976571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ec0c56-20a5-4d87-83f6-1142d5ba8774 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:04.977050 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ec0c56-20a5-4d87-83f6-1142d5ba8774 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:04.977514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ec0c56-20a5-4d87-83f6-1142d5ba8774 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:04.986532 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:04.989131 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ec0c56-20a5-4d87-83f6-1142d5ba8774 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:04.990863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ec0c56-20a5-4d87-83f6-1142d5ba8774 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:05.006190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-b2ec0c56-20a5-4d87-83f6-1142d5ba8774 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:05.006190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:05.006190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:05.006190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:05.006190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:05.006190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:05.006190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:05.006190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:05.006190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:05.006190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:05.006190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:05.006190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:05.006190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:28:05.006190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:28:05.006190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:28:05.006190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:28:05.007469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:05.007469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:05.007469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:05.007469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:05.007469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:05.007469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:05.007469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:05.007469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:05.007469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:05.007469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:05.007469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:05.007469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:05.007469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:05.007469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:05.007469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:05.007469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:05.007469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:28:05.008673 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:28:05.008673 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:05.008673 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:05.008673 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 439/886] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:05.015685 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3514214e-4ef1-4444-898e-dbc614750970 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:05.027874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:05.130206 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3514214e-4ef1-4444-898e-dbc614750970 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:05.167313 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a8ca4c-914a-408f-a512-9c29cde64ab7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:05.167929 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a8ca4c-914a-408f-a512-9c29cde64ab7 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:05.168210 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a8ca4c-914a-408f-a512-9c29cde64ab7 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:05.168485 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a8ca4c-914a-408f-a512-9c29cde64ab7 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:05.169133 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a8ca4c-914a-408f-a512-9c29cde64ab7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:05.169439 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a8ca4c-914a-408f-a512-9c29cde64ab7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:05.170854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a8ca4c-914a-408f-a512-9c29cde64ab7 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:05.183764 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 448/887] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:28:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:05.189158 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ec0c56-20a5-4d87-83f6-1142d5ba8774 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:05.204748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add3b1fd-91c7-4bb4-9944-3a394dc9b5f3 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:05.204748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add3b1fd-91c7-4bb4-9944-3a394dc9b5f3 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:05.204992 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add3b1fd-91c7-4bb4-9944-3a394dc9b5f3 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:05.206359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 449/888] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:28:05.211530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7c5ab0-aafb-4b31-8aae-4bcde4240dfd None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:05.211737 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7c5ab0-aafb-4b31-8aae-4bcde4240dfd None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:05.212041 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7c5ab0-aafb-4b31-8aae-4bcde4240dfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:05.231207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-ee7c5ab0-aafb-4b31-8aae-4bcde4240dfd None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:05.232405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ee7c5ab0-aafb-4b31-8aae-4bcde4240dfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:05.302938 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ec0c56-20a5-4d87-83f6-1142d5ba8774 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:05.308721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 450/889] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:05 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 30 09:28:05.310965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:05.341865 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63839571-0596-4d45-84ab-7bbde7432add None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:05.342924 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63839571-0596-4d45-84ab-7bbde7432add None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:05.343409 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63839571-0596-4d45-84ab-7bbde7432add None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:05.343882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63839571-0596-4d45-84ab-7bbde7432add None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:05.352244 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63839571-0596-4d45-84ab-7bbde7432add None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:05.353929 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63839571-0596-4d45-84ab-7bbde7432add None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:05.366064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-63839571-0596-4d45-84ab-7bbde7432add None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 30 09:28:05.366064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:05.366064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:05.366064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:05.366064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:05.366064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:05.366064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:05.366064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:05.366064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:05.366064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:05.366064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:05.366064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:05.366064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:28:05.366064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:28:05.366064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:28:05.366064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:28:05.367036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:05.367036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:05.367036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:28:05.367036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:28:05.367036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:28:05.367036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:28:05.367036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:05.367036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:05.367036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:05.367036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:05.367036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:05.367036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:05.367036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:05.367036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:05.367036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:05.367036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:05.367036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:05.367036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:05.368013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:05.368013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:05.368013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:28:05.368013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:28:05.368013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:28:05.368013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:28:05.368013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:28:05.368013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:28:05.368013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 30 09:28:05.368013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:05.375015 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 440/890] 10.222.0.39 () {56 vars in 1117 bytes} [Fri Sep 30 09:28:05 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:05.377451 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7c5ab0-aafb-4b31-8aae-4bcde4240dfd None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:05.481181 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7c5ab0-aafb-4b31-8aae-4bcde4240dfd None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:05.516479 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395ce6c9-80a0-48de-bf16-c1ab4642cf56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:05.517488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395ce6c9-80a0-48de-bf16-c1ab4642cf56 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:05.517991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395ce6c9-80a0-48de-bf16-c1ab4642cf56 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:05.518498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395ce6c9-80a0-48de-bf16-c1ab4642cf56 None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:05.519458 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395ce6c9-80a0-48de-bf16-c1ab4642cf56 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:05.519966 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395ce6c9-80a0-48de-bf16-c1ab4642cf56 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:05.521609 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395ce6c9-80a0-48de-bf16-c1ab4642cf56 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:05.545622 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 451/891] 10.222.0.39 () {56 vars in 1149 bytes} [Fri Sep 30 09:28:05 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:05.547442 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63839571-0596-4d45-84ab-7bbde7432add None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:05.590557 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395ce6c9-80a0-48de-bf16-c1ab4642cf56 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:05.606336 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:05.657353 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63839571-0596-4d45-84ab-7bbde7432add None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:05.669767 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:05.695208 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c0fb93-641a-4aab-9c72-24216e792487 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:05.695823 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c0fb93-641a-4aab-9c72-24216e792487 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:05.696105 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c0fb93-641a-4aab-9c72-24216e792487 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:05.696381 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c0fb93-641a-4aab-9c72-24216e792487 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:05.732914 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395ce6c9-80a0-48de-bf16-c1ab4642cf56 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:05.774632 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c0fb93-641a-4aab-9c72-24216e792487 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:05.776131 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c0fb93-641a-4aab-9c72-24216e792487 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:05.778808 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c0cffc-2b65-4558-85d7-dfdcdd21ef52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:05.780828 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c0cffc-2b65-4558-85d7-dfdcdd21ef52 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:05.781316 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c0cffc-2b65-4558-85d7-dfdcdd21ef52 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:05.781789 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c0cffc-2b65-4558-85d7-dfdcdd21ef52 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:05.782563 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c0cffc-2b65-4558-85d7-dfdcdd21ef52 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:05.784193 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c0cffc-2b65-4558-85d7-dfdcdd21ef52 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:05.797014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-03c0cffc-2b65-4558-85d7-dfdcdd21ef52 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 30 09:28:05.797014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:05.797014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:28:05.797014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:28:05.797014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:28:05.797014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:28:05.797014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:28:05.797014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:05.797014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 30 09:28:05.797014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:05.797014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:28:05.797014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:05.797014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:05.797014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:05.797014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:05.797014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:05.797014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:05.798046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:05.798046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:05.798046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:05.798046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:05.798046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:05.798046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:05.798046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:28:05.798046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:28:05.798046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:28:05.798046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:28:05.798046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:05.798046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:05.798046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:05.798046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:05.798046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:05.798046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:05.798934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:05.798934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:05.798934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:05.798934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:05.798934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:05.798934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:05.798934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:05.798934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:05.798934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:05.798934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:05.798934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:28:05.798934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:05.798934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 30 09:28:05.798934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:05.802733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 452/892] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:28:05 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:05.805703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c0cffc-2b65-4558-85d7-dfdcdd21ef52 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:05.844004 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:05.909976 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 441/893] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:28:05 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 30 09:28:05.916087 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c0fb93-641a-4aab-9c72-24216e792487 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:05.935613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c0cffc-2b65-4558-85d7-dfdcdd21ef52 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:05.963015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:05.981797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8ee02d-32bc-4626-bf51-1e8e56b3a142 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:05.982909 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8ee02d-32bc-4626-bf51-1e8e56b3a142 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:05.983854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8ee02d-32bc-4626-bf51-1e8e56b3a142 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:05.984591 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8ee02d-32bc-4626-bf51-1e8e56b3a142 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:05.985649 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8ee02d-32bc-4626-bf51-1e8e56b3a142 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:05.986157 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8ee02d-32bc-4626-bf51-1e8e56b3a142 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:05.988513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8ee02d-32bc-4626-bf51-1e8e56b3a142 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:05.995638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 453/894] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:28:05 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:06.006619 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8ee02d-32bc-4626-bf51-1e8e56b3a142 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:06.041272 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c0fb93-641a-4aab-9c72-24216e792487 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:06.082108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cb1127-da2e-4cb9-ae30-0b5e18a4eb87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:06.082776 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cb1127-da2e-4cb9-ae30-0b5e18a4eb87 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:06.083056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cb1127-da2e-4cb9-ae30-0b5e18a4eb87 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:06.083326 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cb1127-da2e-4cb9-ae30-0b5e18a4eb87 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:06.097156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04cb1127-da2e-4cb9-ae30-0b5e18a4eb87 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:06.098746 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04cb1127-da2e-4cb9-ae30-0b5e18a4eb87 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:06.117337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-04cb1127-da2e-4cb9-ae30-0b5e18a4eb87 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:06.117337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:06.117337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:06.117337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:06.117337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:06.117337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:06.117337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:06.117337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:06.117337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:06.117337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:06.117337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:06.117337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:06.117337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:28:06.117337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:28:06.117337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:28:06.117337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:28:06.118339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:06.118339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:06.118339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:06.118339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:06.118339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:06.118339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:06.118339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:06.118339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:06.118339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:06.118339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:06.118339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:06.118339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:06.118339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:06.118339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:06.118339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:06.118339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:06.118339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:28:06.119356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:28:06.119356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:06.119356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:06.119555 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8ee02d-32bc-4626-bf51-1e8e56b3a142 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:06.122681 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 442/895] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:06.125470 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cb1127-da2e-4cb9-ae30-0b5e18a4eb87 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:06.130166 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:06.159459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4c1c93-2590-45bd-aaa4-a357f3d84abe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:06.160379 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4c1c93-2590-45bd-aaa4-a357f3d84abe None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:06.160666 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4c1c93-2590-45bd-aaa4-a357f3d84abe None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:06.160938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4c1c93-2590-45bd-aaa4-a357f3d84abe None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:06.161560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4c1c93-2590-45bd-aaa4-a357f3d84abe None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:06.161864 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4c1c93-2590-45bd-aaa4-a357f3d84abe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:06.163439 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4c1c93-2590-45bd-aaa4-a357f3d84abe None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:06.169002 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 454/896] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:28:05 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:06.174811 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f74cea-bf1d-4ef0-ad13-a4b6d02c28dc None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:06.175099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f74cea-bf1d-4ef0-ad13-a4b6d02c28dc None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:06.175383 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f74cea-bf1d-4ef0-ad13-a4b6d02c28dc None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:06.178641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 455/897] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:06.178641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f74cea-bf1d-4ef0-ad13-a4b6d02c28dc None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:06.237808 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:06.239947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cb1127-da2e-4cb9-ae30-0b5e18a4eb87 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:06.275082 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c43f8d-082d-4f3c-b6c5-0611de47a197 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:06.275681 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c43f8d-082d-4f3c-b6c5-0611de47a197 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:06.276446 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c43f8d-082d-4f3c-b6c5-0611de47a197 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:06.276718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c43f8d-082d-4f3c-b6c5-0611de47a197 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:06.277344 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c43f8d-082d-4f3c-b6c5-0611de47a197 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:06.277646 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c43f8d-082d-4f3c-b6c5-0611de47a197 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:06.279049 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c43f8d-082d-4f3c-b6c5-0611de47a197 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:06.291450 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 443/898] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:28:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:06.292934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f74cea-bf1d-4ef0-ad13-a4b6d02c28dc None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:06.293394 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c5067a-0d4a-4cc2-8308-497ab3f8412a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:06.293745 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c5067a-0d4a-4cc2-8308-497ab3f8412a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:06.294085 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c5067a-0d4a-4cc2-8308-497ab3f8412a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:06.311401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-38c5067a-0d4a-4cc2-8308-497ab3f8412a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:06.312703 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-38c5067a-0d4a-4cc2-8308-497ab3f8412a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:06.329590 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b590cf32-91af-4f59-a796-5b0b52b40b77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:06.330310 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b590cf32-91af-4f59-a796-5b0b52b40b77 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:06.330625 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b590cf32-91af-4f59-a796-5b0b52b40b77 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:06.330897 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b590cf32-91af-4f59-a796-5b0b52b40b77 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:06.331585 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b590cf32-91af-4f59-a796-5b0b52b40b77 None admin] RBAC: Adding query filter params (interface=public, service_id=d7f5b065b8f641cf8939444c7065959a, region_id=RegionOne) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:06.331893 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b590cf32-91af-4f59-a796-5b0b52b40b77 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:06.333264 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b590cf32-91af-4f59-a796-5b0b52b40b77 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:06.338808 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 456/899] 10.222.0.39 () {56 vars in 1251 bytes} [Fri Sep 30 09:28:06 2022] GET /identity/v3/endpoints?service_id=d7f5b065b8f641cf8939444c7065959a&interface=public®ion_id=RegionOne => generated 194 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:06.343236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b590cf32-91af-4f59-a796-5b0b52b40b77 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:06.373184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:06.382758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 444/900] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:06 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:06.383573 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c5067a-0d4a-4cc2-8308-497ab3f8412a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:06.475805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b590cf32-91af-4f59-a796-5b0b52b40b77 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:06.489939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:06.502094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c5067a-0d4a-4cc2-8308-497ab3f8412a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:06.511283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19882d9-2b50-4657-90ae-1c167b44498f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:06.511919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19882d9-2b50-4657-90ae-1c167b44498f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:06.512198 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19882d9-2b50-4657-90ae-1c167b44498f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:06.512471 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19882d9-2b50-4657-90ae-1c167b44498f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:06.519774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19882d9-2b50-4657-90ae-1c167b44498f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:06.521537 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19882d9-2b50-4657-90ae-1c167b44498f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:06.533721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-d19882d9-2b50-4657-90ae-1c167b44498f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:28:06.533721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:06.533721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:06.533721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:06.533721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:06.533721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:06.533721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:06.533721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:06.533721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:06.533721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:06.533721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:06.533721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:06.533721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:28:06.533721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:28:06.533721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:28:06.534846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:28:06.534846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:06.534846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:06.534846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:06.534846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:06.534846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:06.534846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:06.534846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:06.534846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:06.534846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:06.534846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:06.534846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:06.534846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:06.534846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:06.534846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:06.534846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:06.534846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:06.535962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:28:06.535962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:28:06.535962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:28:06.535962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:28:06.535962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:28:06.535962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:28:06.535962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:28:06.535962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:06.538134 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 457/901] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:28:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:06.544341 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19882d9-2b50-4657-90ae-1c167b44498f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:06.549954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15695d57-378c-4d2f-8c85-171fcafafb6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:06.550793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15695d57-378c-4d2f-8c85-171fcafafb6a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:06.551235 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15695d57-378c-4d2f-8c85-171fcafafb6a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:06.551652 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15695d57-378c-4d2f-8c85-171fcafafb6a None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:06.552391 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15695d57-378c-4d2f-8c85-171fcafafb6a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:06.553884 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15695d57-378c-4d2f-8c85-171fcafafb6a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:06.564409 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 445/902] 10.222.0.39 () {56 vars in 1087 bytes} [Fri Sep 30 09:28:06 2022] GET /identity/v3/services => generated 1268 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:28:06.656093 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15695d57-378c-4d2f-8c85-171fcafafb6a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:06.664599 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19882d9-2b50-4657-90ae-1c167b44498f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:06.703377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d37148-14b9-4a64-9fd4-02156ce60c8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:06.704465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d37148-14b9-4a64-9fd4-02156ce60c8b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:06.704967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d37148-14b9-4a64-9fd4-02156ce60c8b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:06.705434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d37148-14b9-4a64-9fd4-02156ce60c8b None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:06.706326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d37148-14b9-4a64-9fd4-02156ce60c8b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:06.706886 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d37148-14b9-4a64-9fd4-02156ce60c8b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:06.708571 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d37148-14b9-4a64-9fd4-02156ce60c8b None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:06.723609 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 458/903] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:28:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:06.730743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d37148-14b9-4a64-9fd4-02156ce60c8b None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:06.740804 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:06.767902 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:06.796036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15695d57-378c-4d2f-8c85-171fcafafb6a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:06.838262 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741a1fb4-2898-4766-b793-cf98853a4cb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:06.839568 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741a1fb4-2898-4766-b793-cf98853a4cb2 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:06.840075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741a1fb4-2898-4766-b793-cf98853a4cb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:06.841037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741a1fb4-2898-4766-b793-cf98853a4cb2 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:06.842200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741a1fb4-2898-4766-b793-cf98853a4cb2 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:06.844738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741a1fb4-2898-4766-b793-cf98853a4cb2 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:06.846857 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d37148-14b9-4a64-9fd4-02156ce60c8b None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:06.884064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e21f4f-6328-43f5-bd01-f7ed73c44986 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:06.884709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e21f4f-6328-43f5-bd01-f7ed73c44986 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:06.884984 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e21f4f-6328-43f5-bd01-f7ed73c44986 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:06.885302 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e21f4f-6328-43f5-bd01-f7ed73c44986 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:06.886014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e21f4f-6328-43f5-bd01-f7ed73c44986 None admin] RBAC: Adding query filter params (role.id=fa71d6ff1b04452c98c5830f178feb1b, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=7d6e0545a40a436d8c63820b795551a9) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:06.886317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e21f4f-6328-43f5-bd01-f7ed73c44986 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:06.887777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e21f4f-6328-43f5-bd01-f7ed73c44986 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:06.893551 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 459/904] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:28:06 2022] GET /identity/v3/role_assignments?user.id=7d6e0545a40a436d8c63820b795551a9&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=fa71d6ff1b04452c98c5830f178feb1b => generated 259 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:06.923168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 446/905] 10.222.0.39 () {60 vars in 1141 bytes} [Fri Sep 30 09:28:06 2022] POST /identity/v3/services => generated 241 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:28:07.051690 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf33619-d08a-40eb-a657-4363c9b88998 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:07.051870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf33619-d08a-40eb-a657-4363c9b88998 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:07.052156 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf33619-d08a-40eb-a657-4363c9b88998 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:07.053908 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 460/906] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:07.060084 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db732b5-b5f8-4c99-92c7-dfcd0d058de5 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:07.060387 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db732b5-b5f8-4c99-92c7-dfcd0d058de5 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:07.060798 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db732b5-b5f8-4c99-92c7-dfcd0d058de5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:07.081344 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-2db732b5-b5f8-4c99-92c7-dfcd0d058de5 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:07.082616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2db732b5-b5f8-4c99-92c7-dfcd0d058de5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:07.162571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 447/907] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:07 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:07.435050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf33619-d08a-40eb-a657-4363c9b88998 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:07.544386 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:07.551569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf33619-d08a-40eb-a657-4363c9b88998 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:07.586346 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c58da8a-995c-4980-a7c9-c0bab9255ec0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:07.586999 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c58da8a-995c-4980-a7c9-c0bab9255ec0 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:07.587295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c58da8a-995c-4980-a7c9-c0bab9255ec0 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:07.587592 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c58da8a-995c-4980-a7c9-c0bab9255ec0 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:07.588189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c58da8a-995c-4980-a7c9-c0bab9255ec0 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:07.590183 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c58da8a-995c-4980-a7c9-c0bab9255ec0 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:07.599658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-7c58da8a-995c-4980-a7c9-c0bab9255ec0 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 30 09:28:07.599658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:07.599658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:28:07.599658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:28:07.599658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:28:07.599658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:28:07.599658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:28:07.599658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:07.599658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 30 09:28:07.599658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:07.599658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:28:07.599658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:07.599658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:07.599658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:07.599658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:07.599658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:07.600598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:07.600598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:07.600598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:07.600598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:07.600598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:07.600598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:07.600598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:07.600598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:28:07.600598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:28:07.600598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:28:07.600598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:28:07.600598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:07.600598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:07.600598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:07.600598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:07.600598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:07.600598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:07.601527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:07.601527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:07.601527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:07.601527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:07.601527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:07.601527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:07.601527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:07.601527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:07.601527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:07.601527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:07.601527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:28:07.601527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:07.601527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 30 09:28:07.601527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:07.601527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 461/908] 10.222.0.39 () {56 vars in 1132 bytes} [Fri Sep 30 09:28:07 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:07.606828 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db732b5-b5f8-4c99-92c7-dfcd0d058de5 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:07.655218 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:07.711587 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db732b5-b5f8-4c99-92c7-dfcd0d058de5 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:07.746548 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6f8dd8-5ad0-417d-a8e3-4850913c83f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:07.747281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6f8dd8-5ad0-417d-a8e3-4850913c83f6 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:07.747565 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6f8dd8-5ad0-417d-a8e3-4850913c83f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:07.747888 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6f8dd8-5ad0-417d-a8e3-4850913c83f6 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:07.748560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6f8dd8-5ad0-417d-a8e3-4850913c83f6 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:07.748867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6f8dd8-5ad0-417d-a8e3-4850913c83f6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:07.750226 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6f8dd8-5ad0-417d-a8e3-4850913c83f6 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:07.755257 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 448/909] 10.222.0.39 () {56 vars in 1126 bytes} [Fri Sep 30 09:28:07 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:07.761775 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c58da8a-995c-4980-a7c9-c0bab9255ec0 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:07.875246 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c58da8a-995c-4980-a7c9-c0bab9255ec0 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:07.911152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868c5274-eef4-447b-a259-4880d1222271 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:07.911772 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868c5274-eef4-447b-a259-4880d1222271 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:07.912052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868c5274-eef4-447b-a259-4880d1222271 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:07.912330 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868c5274-eef4-447b-a259-4880d1222271 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:07.912966 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868c5274-eef4-447b-a259-4880d1222271 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:07.913273 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868c5274-eef4-447b-a259-4880d1222271 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:07.914720 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868c5274-eef4-447b-a259-4880d1222271 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:07.920968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 462/910] 10.222.0.39 () {56 vars in 1126 bytes} [Fri Sep 30 09:28:07 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:07.926955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6f8dd8-5ad0-417d-a8e3-4850913c83f6 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:08.028252 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6f8dd8-5ad0-417d-a8e3-4850913c83f6 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:08.038191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:08.063025 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94edb1c-32d2-44ad-a8cb-5f8df90df215 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:08.063629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94edb1c-32d2-44ad-a8cb-5f8df90df215 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:08.069058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94edb1c-32d2-44ad-a8cb-5f8df90df215 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:08.069058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94edb1c-32d2-44ad-a8cb-5f8df90df215 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:08.069058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94edb1c-32d2-44ad-a8cb-5f8df90df215 None admin] RBAC: Adding query filter params (interface=public, service_id=10530be545eb42c898dd4185001b85c0, region_id=RegionOne) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:08.069058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94edb1c-32d2-44ad-a8cb-5f8df90df215 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:08.069058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94edb1c-32d2-44ad-a8cb-5f8df90df215 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:08.074607 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 449/911] 10.222.0.39 () {56 vars in 1251 bytes} [Fri Sep 30 09:28:07 2022] GET /identity/v3/endpoints?service_id=10530be545eb42c898dd4185001b85c0&interface=public®ion_id=RegionOne => generated 194 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:08.080260 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868c5274-eef4-447b-a259-4880d1222271 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:08.110707 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:08.190292 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868c5274-eef4-447b-a259-4880d1222271 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:08.229441 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a0d8d0-1196-4c41-957d-7e54248d0b3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:08.230055 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a0d8d0-1196-4c41-957d-7e54248d0b3d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:08.230334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a0d8d0-1196-4c41-957d-7e54248d0b3d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:08.230618 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a0d8d0-1196-4c41-957d-7e54248d0b3d None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:08.231230 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a0d8d0-1196-4c41-957d-7e54248d0b3d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:08.232612 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a0d8d0-1196-4c41-957d-7e54248d0b3d None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:08.239169 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 463/912] 10.222.0.39 () {56 vars in 1087 bytes} [Fri Sep 30 09:28:08 2022] GET /identity/v3/services => generated 1497 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:28:08.879873 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37b7297-5225-49ac-8b64-505c37fb4097 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:08.880753 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37b7297-5225-49ac-8b64-505c37fb4097 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:08.880931 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37b7297-5225-49ac-8b64-505c37fb4097 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:08.882647 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 450/913] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:08.888675 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8617d50b-7c7b-4907-934f-6a38ecfe6db9 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:08.889117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8617d50b-7c7b-4907-934f-6a38ecfe6db9 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:08.889426 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8617d50b-7c7b-4907-934f-6a38ecfe6db9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:08.911526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-8617d50b-7c7b-4907-934f-6a38ecfe6db9 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:08.912947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-8617d50b-7c7b-4907-934f-6a38ecfe6db9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:08.985617 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 464/914] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:08 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:09.144457 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472c955d-d026-42a5-9f3f-efaa48620d52 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:09.144648 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472c955d-d026-42a5-9f3f-efaa48620d52 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:09.144925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472c955d-d026-42a5-9f3f-efaa48620d52 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:09.146786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 451/915] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:09.153083 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b7b878-55eb-4681-9d1e-13f1474ed219 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:09.153786 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b7b878-55eb-4681-9d1e-13f1474ed219 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:09.154266 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b7b878-55eb-4681-9d1e-13f1474ed219 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:09.172991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-a3b7b878-55eb-4681-9d1e-13f1474ed219 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:09.174954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a3b7b878-55eb-4681-9d1e-13f1474ed219 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:09.190811 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:09.238131 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83c1023-0607-4957-9782-43acac7b05ef None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:09.238419 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83c1023-0607-4957-9782-43acac7b05ef None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:09.238746 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83c1023-0607-4957-9782-43acac7b05ef None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:09.240335 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 452/916] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:09.246571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9281d4d7-2b12-4eab-9b6e-2bd99238bf88 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:09.246899 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9281d4d7-2b12-4eab-9b6e-2bd99238bf88 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:09.247177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9281d4d7-2b12-4eab-9b6e-2bd99238bf88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:09.258614 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 465/917] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:09 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:09.271584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9281d4d7-2b12-4eab-9b6e-2bd99238bf88 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:09.272763 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9281d4d7-2b12-4eab-9b6e-2bd99238bf88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:09.279888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b7b878-55eb-4681-9d1e-13f1474ed219 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:09.358678 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 453/918] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:09 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:09.399744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b7b878-55eb-4681-9d1e-13f1474ed219 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:09.435475 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00dceae5-c190-46b8-b4a4-2687821d8000 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:09.436434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dceae5-c190-46b8-b4a4-2687821d8000 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:09.436944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dceae5-c190-46b8-b4a4-2687821d8000 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:09.437433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dceae5-c190-46b8-b4a4-2687821d8000 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:09.438218 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00dceae5-c190-46b8-b4a4-2687821d8000 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:09.439882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00dceae5-c190-46b8-b4a4-2687821d8000 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:09.449712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-00dceae5-c190-46b8-b4a4-2687821d8000 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 30 09:28:09.449712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:09.449712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:28:09.449712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:28:09.449712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:28:09.449712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:28:09.449712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:28:09.449712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:09.449712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 30 09:28:09.449712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:09.449712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:28:09.449712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:09.449712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:09.449712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:09.449712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:09.449712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:09.449712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:09.450833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:09.450833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:09.450833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:09.450833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:09.450833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:09.450833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:09.450833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:28:09.450833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:28:09.450833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:28:09.450833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:28:09.450833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:09.450833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:09.450833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:09.450833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:09.450833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:09.450833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:09.451784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:09.451784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:09.451784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:09.451784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:09.451784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:09.451784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:09.451784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:09.451784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:09.451784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:09.451784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:09.451784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:28:09.451784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:09.451784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 30 09:28:09.451784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:09.454541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 466/919] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:28:09 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:09.460610 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9281d4d7-2b12-4eab-9b6e-2bd99238bf88 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:09.520223 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00dceae5-c190-46b8-b4a4-2687821d8000 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:09.536254 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:09.576614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9281d4d7-2b12-4eab-9b6e-2bd99238bf88 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:09.578123 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:09.612230 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a939a64-3eee-4852-9a2f-23ffb561fdc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:09.612864 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a939a64-3eee-4852-9a2f-23ffb561fdc7 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:09.613138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a939a64-3eee-4852-9a2f-23ffb561fdc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:09.613407 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a939a64-3eee-4852-9a2f-23ffb561fdc7 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:09.614337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a939a64-3eee-4852-9a2f-23ffb561fdc7 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:09.614337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a939a64-3eee-4852-9a2f-23ffb561fdc7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:09.615735 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a939a64-3eee-4852-9a2f-23ffb561fdc7 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:09.628891 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 454/920] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:28:09 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:09.630134 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a939a64-3eee-4852-9a2f-23ffb561fdc7 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:09.650252 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00dceae5-c190-46b8-b4a4-2687821d8000 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:09.691329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766bc5b2-7f66-43a1-86e9-2045fe86c76c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:09.691955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766bc5b2-7f66-43a1-86e9-2045fe86c76c None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:09.692241 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766bc5b2-7f66-43a1-86e9-2045fe86c76c None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:09.692529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766bc5b2-7f66-43a1-86e9-2045fe86c76c None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:09.702548 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766bc5b2-7f66-43a1-86e9-2045fe86c76c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:28:09.707541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-766bc5b2-7f66-43a1-86e9-2045fe86c76c None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 30 09:28:09.707541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:09.707541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:09.707541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:09.707541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:09.707541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:09.707541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:09.707541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:09.707541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:09.707541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:09.707541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:09.707541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:09.707541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:28:09.707541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:28:09.707541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:28:09.707541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:28:09.708513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:28:09.708513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:28:09.708513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:28:09.708513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:28:09.708513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:09.708513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:09.708513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:09.708513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:09.708513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:09.708513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:09.708513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:09.708513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:09.708513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:09.708513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:09.708513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:09.708513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:09.709369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:09.709369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:09.709369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:09.709369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:09.709369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:28:09.709369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:28:09.709369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:28:09.709369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:28:09.709369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:28:09.709369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:28:09.709369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 30 09:28:09.709369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:09.709369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 467/921] 10.222.0.39 () {56 vars in 1120 bytes} [Fri Sep 30 09:28:09 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:09.710116 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766bc5b2-7f66-43a1-86e9-2045fe86c76c None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:09.740034 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a939a64-3eee-4852-9a2f-23ffb561fdc7 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:09.776030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80aff811-50c9-4f00-b95c-607dd6142e9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:09.776801 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aff811-50c9-4f00-b95c-607dd6142e9e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:09.777083 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aff811-50c9-4f00-b95c-607dd6142e9e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:09.777355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aff811-50c9-4f00-b95c-607dd6142e9e None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:09.777972 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80aff811-50c9-4f00-b95c-607dd6142e9e None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:09.778276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80aff811-50c9-4f00-b95c-607dd6142e9e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:09.779775 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80aff811-50c9-4f00-b95c-607dd6142e9e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:09.780637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:09.785560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 455/922] 10.222.0.39 () {56 vars in 1114 bytes} [Fri Sep 30 09:28:09 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:09.786422 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80aff811-50c9-4f00-b95c-607dd6142e9e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:09.815554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766bc5b2-7f66-43a1-86e9-2045fe86c76c None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:09.819217 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:09.849046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32a246d-a05c-469a-9403-e1e6c659d98b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:09.849670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32a246d-a05c-469a-9403-e1e6c659d98b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:09.849973 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32a246d-a05c-469a-9403-e1e6c659d98b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:09.850248 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32a246d-a05c-469a-9403-e1e6c659d98b None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:09.850858 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32a246d-a05c-469a-9403-e1e6c659d98b None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:09.852229 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32a246d-a05c-469a-9403-e1e6c659d98b None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:09.861505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-d32a246d-a05c-469a-9403-e1e6c659d98b None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 30 09:28:09.861505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:09.861505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:28:09.861505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:28:09.861505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:28:09.861505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:28:09.861505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:28:09.861505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:09.861505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 30 09:28:09.861505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:09.861505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:28:09.861505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:09.861505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:09.861505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:09.861505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:09.861505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:09.861505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:09.862529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:09.862529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:09.862529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:09.862529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:09.862529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:09.862529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:09.862529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:28:09.862529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:28:09.862529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:28:09.862529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:28:09.862529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:09.862529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:09.862529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:09.862529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:09.862529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:09.862529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:09.863491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:09.863491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:09.863491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:09.863491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:09.863491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:09.863491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:09.863491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:09.863491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:09.863491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:09.863491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:09.863491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:28:09.863491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:09.863491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 30 09:28:09.863491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:09.863491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 468/923] 10.222.0.39 () {56 vars in 1126 bytes} [Fri Sep 30 09:28:09 2022] GET /identity/v3/services/object-store => generated 93 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:09.864313 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32a246d-a05c-469a-9403-e1e6c659d98b None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:09.898977 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80aff811-50c9-4f00-b95c-607dd6142e9e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:09.935897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7feb203d-38e4-4960-8b45-96c6b9a04c34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:09.936407 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feb203d-38e4-4960-8b45-96c6b9a04c34 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:09.936685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feb203d-38e4-4960-8b45-96c6b9a04c34 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:09.936956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feb203d-38e4-4960-8b45-96c6b9a04c34 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:09.937609 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7feb203d-38e4-4960-8b45-96c6b9a04c34 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:09.937920 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7feb203d-38e4-4960-8b45-96c6b9a04c34 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:09.939338 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7feb203d-38e4-4960-8b45-96c6b9a04c34 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:09.949809 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 456/924] 10.222.0.39 () {56 vars in 1115 bytes} [Fri Sep 30 09:28:09 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:09.951186 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7feb203d-38e4-4960-8b45-96c6b9a04c34 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:09.980065 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32a246d-a05c-469a-9403-e1e6c659d98b None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:10.018158 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bab94da-7990-401c-97fc-09aa03946531 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:10.019311 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bab94da-7990-401c-97fc-09aa03946531 None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:10.019792 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bab94da-7990-401c-97fc-09aa03946531 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:10.020752 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bab94da-7990-401c-97fc-09aa03946531 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:10.021890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bab94da-7990-401c-97fc-09aa03946531 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:10.024442 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bab94da-7990-401c-97fc-09aa03946531 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:10.031329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:10.045527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:10.053807 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7feb203d-38e4-4960-8b45-96c6b9a04c34 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:10.086464 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a1132a-aa75-44da-956b-6c28bc7f5fdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:10.087137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a1132a-aa75-44da-956b-6c28bc7f5fdc None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:10.087415 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a1132a-aa75-44da-956b-6c28bc7f5fdc None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:10.088590 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a1132a-aa75-44da-956b-6c28bc7f5fdc None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:10.088812 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a1132a-aa75-44da-956b-6c28bc7f5fdc None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:10.089096 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a1132a-aa75-44da-956b-6c28bc7f5fdc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:10.090441 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a1132a-aa75-44da-956b-6c28bc7f5fdc None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:10.095111 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 457/925] 10.222.0.39 () {56 vars in 1122 bytes} [Fri Sep 30 09:28:09 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:10.095697 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a1132a-aa75-44da-956b-6c28bc7f5fdc None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:10.122621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 469/926] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:28:09 2022] POST /identity/v3/endpoints => generated 346 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:28:10.123975 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bab94da-7990-401c-97fc-09aa03946531 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:10.224053 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a1132a-aa75-44da-956b-6c28bc7f5fdc None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:10.276150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2031b64b-5134-4842-b6aa-bbc2421fd2fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:10.277231 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2031b64b-5134-4842-b6aa-bbc2421fd2fe None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:10.277736 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2031b64b-5134-4842-b6aa-bbc2421fd2fe None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:10.278897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2031b64b-5134-4842-b6aa-bbc2421fd2fe None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:10.295092 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2031b64b-5134-4842-b6aa-bbc2421fd2fe None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:10.296724 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2031b64b-5134-4842-b6aa-bbc2421fd2fe None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:10.297700 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bab94da-7990-401c-97fc-09aa03946531 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:10.311875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-2031b64b-5134-4842-b6aa-bbc2421fd2fe None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:10.311875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:10.311875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:10.311875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:10.311875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:10.311875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:10.311875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:10.311875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:10.311875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:10.311875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:10.311875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:10.311875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:10.311875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:28:10.311875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:28:10.311875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:28:10.311875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:28:10.312908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:10.312908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:10.312908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:10.312908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:10.312908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:10.312908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:10.312908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:10.312908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:10.312908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:10.312908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:10.312908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:10.312908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:10.312908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:10.312908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:10.312908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:10.312908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:10.312908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:28:10.313835 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:28:10.313835 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:10.313835 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:10.313835 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 458/927] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:10.320324 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2031b64b-5134-4842-b6aa-bbc2421fd2fe None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:10.327947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:10.347457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f981a1-6417-4be3-a076-437c650fd3db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:10.348379 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f981a1-6417-4be3-a076-437c650fd3db None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:10.348818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f981a1-6417-4be3-a076-437c650fd3db None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:10.349256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f981a1-6417-4be3-a076-437c650fd3db None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:10.349916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f981a1-6417-4be3-a076-437c650fd3db None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:10.350227 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f981a1-6417-4be3-a076-437c650fd3db None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:10.351730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f981a1-6417-4be3-a076-437c650fd3db None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:10.365412 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 470/928] 10.222.0.39 () {56 vars in 1122 bytes} [Fri Sep 30 09:28:10 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:10.366739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f981a1-6417-4be3-a076-437c650fd3db None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:10.426899 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:10.453803 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2031b64b-5134-4842-b6aa-bbc2421fd2fe None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:10.473495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f981a1-6417-4be3-a076-437c650fd3db None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:10.499627 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17abab27-6e6c-469c-9bae-3a187f41e06b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:10.500561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17abab27-6e6c-469c-9bae-3a187f41e06b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:10.500851 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17abab27-6e6c-469c-9bae-3a187f41e06b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:10.501402 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17abab27-6e6c-469c-9bae-3a187f41e06b None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:10.502050 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17abab27-6e6c-469c-9bae-3a187f41e06b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:10.502512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17abab27-6e6c-469c-9bae-3a187f41e06b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:10.504543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17abab27-6e6c-469c-9bae-3a187f41e06b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:10.507955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766b394a-8d95-4e74-a8ac-15e81ed2134c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:10.508578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766b394a-8d95-4e74-a8ac-15e81ed2134c None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:10.508870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766b394a-8d95-4e74-a8ac-15e81ed2134c None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:10.509128 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766b394a-8d95-4e74-a8ac-15e81ed2134c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:10.509813 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766b394a-8d95-4e74-a8ac-15e81ed2134c None admin] RBAC: Adding query filter params (interface=public, service_id=ecac99253aaa43bfa5c89f8a9e3e5288, region_id=RegionOne) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:10.510666 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766b394a-8d95-4e74-a8ac-15e81ed2134c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:10.512066 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766b394a-8d95-4e74-a8ac-15e81ed2134c None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:10.522250 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 471/929] 10.222.0.39 () {56 vars in 1251 bytes} [Fri Sep 30 09:28:10 2022] GET /identity/v3/endpoints?service_id=ecac99253aaa43bfa5c89f8a9e3e5288&interface=public®ion_id=RegionOne => generated 194 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:10.522412 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 459/930] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:28:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:10.523472 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73aab207-2874-4f54-bac1-e657aa0018ac None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:10.523584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17abab27-6e6c-469c-9bae-3a187f41e06b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:10.523795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73aab207-2874-4f54-bac1-e657aa0018ac None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:10.524033 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73aab207-2874-4f54-bac1-e657aa0018ac None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:10.525425 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 472/931] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:10 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:10.530205 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73aab207-2874-4f54-bac1-e657aa0018ac None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:10.595506 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:10.636648 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17abab27-6e6c-469c-9bae-3a187f41e06b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:10.639582 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73aab207-2874-4f54-bac1-e657aa0018ac None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:10.660541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:10.675769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4645d1-5480-4ff1-ba5a-e3914efb038c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:10.675969 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e18d30-2d5b-4019-846e-ba59dfa3cb65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:10.676401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4645d1-5480-4ff1-ba5a-e3914efb038c None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:10.676586 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e18d30-2d5b-4019-846e-ba59dfa3cb65 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:10.676692 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4645d1-5480-4ff1-ba5a-e3914efb038c None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:10.676883 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e18d30-2d5b-4019-846e-ba59dfa3cb65 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:10.676980 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4645d1-5480-4ff1-ba5a-e3914efb038c None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:10.677144 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e18d30-2d5b-4019-846e-ba59dfa3cb65 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:10.677706 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e18d30-2d5b-4019-846e-ba59dfa3cb65 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:10.679114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e18d30-2d5b-4019-846e-ba59dfa3cb65 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:10.684673 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4645d1-5480-4ff1-ba5a-e3914efb038c None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:10.686431 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4645d1-5480-4ff1-ba5a-e3914efb038c None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:10.690795 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 460/932] 10.222.0.39 () {56 vars in 1087 bytes} [Fri Sep 30 09:28:10 2022] GET /identity/v3/services => generated 1497 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:28:10.692185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d8de03-1709-42f9-9300-701b809cfbda None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:10.692477 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d8de03-1709-42f9-9300-701b809cfbda None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:10.692755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d8de03-1709-42f9-9300-701b809cfbda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:10.701840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-4b4645d1-5480-4ff1-ba5a-e3914efb038c None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 30 09:28:10.701840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:10.701840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:10.701840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:10.701840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:10.701840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:10.701840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:10.701840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:10.701840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:10.701840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:10.701840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:10.701840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:10.701840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:28:10.701840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:28:10.701840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:28:10.701840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:28:10.703267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:10.703267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:10.703267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:28:10.703267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:28:10.703267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:28:10.703267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:28:10.703267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:10.703267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:10.703267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:10.703267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:10.703267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:10.703267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:10.703267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:10.703267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:10.703267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:10.703267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:10.703267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:10.703267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:10.704306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:10.704306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:10.704306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:28:10.704306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:28:10.704306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:28:10.704306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:28:10.704306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:28:10.704306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:28:10.704306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 30 09:28:10.704306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:10.704306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 473/933] 10.222.0.39 () {56 vars in 1117 bytes} [Fri Sep 30 09:28:10 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:10.711664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4645d1-5480-4ff1-ba5a-e3914efb038c None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:10.723734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-20d8de03-1709-42f9-9300-701b809cfbda None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:10.724982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-20d8de03-1709-42f9-9300-701b809cfbda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:10.802882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 461/934] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:10 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:10.839436 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4645d1-5480-4ff1-ba5a-e3914efb038c None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:10.878038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43907f1-1926-44f4-b549-208cd286f9e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:10.878767 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43907f1-1926-44f4-b549-208cd286f9e6 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:10.879054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43907f1-1926-44f4-b549-208cd286f9e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:10.879331 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43907f1-1926-44f4-b549-208cd286f9e6 None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:10.880073 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43907f1-1926-44f4-b549-208cd286f9e6 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:10.880763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43907f1-1926-44f4-b549-208cd286f9e6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:10.882224 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43907f1-1926-44f4-b549-208cd286f9e6 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:10.903601 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 474/935] 10.222.0.39 () {56 vars in 1149 bytes} [Fri Sep 30 09:28:10 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:10.909245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d8de03-1709-42f9-9300-701b809cfbda None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:10.976476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:11.022849 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d8de03-1709-42f9-9300-701b809cfbda None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:11.059689 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8be61d4-1550-413d-9930-045eefd4aff1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:11.060332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8be61d4-1550-413d-9930-045eefd4aff1 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:11.060613 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8be61d4-1550-413d-9930-045eefd4aff1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:11.060890 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8be61d4-1550-413d-9930-045eefd4aff1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:11.072008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8be61d4-1550-413d-9930-045eefd4aff1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:11.074016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8be61d4-1550-413d-9930-045eefd4aff1 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:11.085495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43907f1-1926-44f4-b549-208cd286f9e6 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:11.090728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-d8be61d4-1550-413d-9930-045eefd4aff1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:11.090728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:11.090728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:11.090728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:11.090728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:11.090728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:11.090728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:11.090728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:11.090728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:11.090728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:11.090728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:11.090728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:11.090728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:28:11.090728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:28:11.090728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:28:11.090728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:28:11.091696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:11.091696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:11.091696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:11.091696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:11.091696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:11.091696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:11.091696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:11.091696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:11.091696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:11.091696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:11.091696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:11.091696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:11.091696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:11.091696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:11.091696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:11.091696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:11.091696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:28:11.092642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:28:11.092642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:11.092642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:11.096513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 462/936] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:11.102347 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8be61d4-1550-413d-9930-045eefd4aff1 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:11.147303 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:11.226166 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8be61d4-1550-413d-9930-045eefd4aff1 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:11.230482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43907f1-1926-44f4-b549-208cd286f9e6 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:11.238448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:11.265941 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf64877-5c8f-4703-b8be-f8de9bdfd794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:11.270732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf64877-5c8f-4703-b8be-f8de9bdfd794 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:11.270732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf64877-5c8f-4703-b8be-f8de9bdfd794 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:11.270732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf64877-5c8f-4703-b8be-f8de9bdfd794 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:11.270732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf64877-5c8f-4703-b8be-f8de9bdfd794 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:11.270732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf64877-5c8f-4703-b8be-f8de9bdfd794 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:11.271092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b5a691-70fc-4873-ad16-12b728c4da1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:11.271092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b5a691-70fc-4873-ad16-12b728c4da1e None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:11.271092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b5a691-70fc-4873-ad16-12b728c4da1e None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:11.271303 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf64877-5c8f-4703-b8be-f8de9bdfd794 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:11.271371 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b5a691-70fc-4873-ad16-12b728c4da1e None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:11.271371 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b5a691-70fc-4873-ad16-12b728c4da1e None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:11.272846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b5a691-70fc-4873-ad16-12b728c4da1e None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:11.289751 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 463/937] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:28:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:11.289939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-04b5a691-70fc-4873-ad16-12b728c4da1e None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 30 09:28:11.289939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:11.289939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:28:11.289939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:28:11.289939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:28:11.289939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:28:11.289939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:28:11.289939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:11.289939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 30 09:28:11.289939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:11.289939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:28:11.289939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:11.289939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:11.289939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:11.289939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:11.289939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:11.290883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:11.290883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:11.290883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:11.290883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:11.290883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:11.290883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:11.290883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:11.290883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:28:11.290883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:28:11.290883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:28:11.290883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:28:11.290883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:11.290883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:11.290883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:11.290883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:11.290883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:11.290883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:11.292378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:11.292378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:11.292378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:11.292378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:11.292378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:11.292378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:11.292378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:11.292378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:11.292378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:11.292378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:11.292378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:28:11.292378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:11.292378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 30 09:28:11.292378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:11.292378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 475/938] 10.222.0.39 () {56 vars in 1132 bytes} [Fri Sep 30 09:28:11 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:11.295287 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b5a691-70fc-4873-ad16-12b728c4da1e None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:11.296126 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf64877-5c8f-4703-b8be-f8de9bdfd794 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:11.403940 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b5a691-70fc-4873-ad16-12b728c4da1e None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:11.405897 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:11.414953 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf64877-5c8f-4703-b8be-f8de9bdfd794 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:11.443855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab10b8a2-4221-4d17-b40e-389acc39c3cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:11.444855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab10b8a2-4221-4d17-b40e-389acc39c3cd None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:11.445401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab10b8a2-4221-4d17-b40e-389acc39c3cd None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:11.445964 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab10b8a2-4221-4d17-b40e-389acc39c3cd None admin] PATH_INFO: `/v3/projects/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:11.453895 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56702192-2b69-4a38-909e-25d470d88e98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:11.454511 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56702192-2b69-4a38-909e-25d470d88e98 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:11.454741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab10b8a2-4221-4d17-b40e-389acc39c3cd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:11.454845 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56702192-2b69-4a38-909e-25d470d88e98 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:11.455092 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56702192-2b69-4a38-909e-25d470d88e98 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:11.456299 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56702192-2b69-4a38-909e-25d470d88e98 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:11.456608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56702192-2b69-4a38-909e-25d470d88e98 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:11.457523 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab10b8a2-4221-4d17-b40e-389acc39c3cd None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:11.457969 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56702192-2b69-4a38-909e-25d470d88e98 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:11.468957 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 464/939] 10.222.0.39 () {56 vars in 1126 bytes} [Fri Sep 30 09:28:11 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:11.471561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56702192-2b69-4a38-909e-25d470d88e98 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:11.472890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-ab10b8a2-4221-4d17-b40e-389acc39c3cd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:28:11.472890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:11.472890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:11.472890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:11.472890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:11.472890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:11.472890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:11.472890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:11.472890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:11.472890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:11.472890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:11.472890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:11.472890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:28:11.472890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:28:11.472890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:28:11.473773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:28:11.473773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:11.473773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:11.473773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:11.473773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:11.473773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:11.473773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:11.473773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:11.473773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:11.473773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:11.473773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:11.473773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:11.473773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:11.473773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:11.473773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:11.473773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:11.473773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:11.474714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:28:11.474714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:28:11.474714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:28:11.474714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:28:11.474714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:28:11.474714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:28:11.474714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:28:11.474714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:11.474714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 476/940] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:28:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:11.482469 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab10b8a2-4221-4d17-b40e-389acc39c3cd None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:11.515476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:11.591721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab10b8a2-4221-4d17-b40e-389acc39c3cd None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:11.594119 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56702192-2b69-4a38-909e-25d470d88e98 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:11.630796 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e01b8f-4ea7-4dc7-a4c2-f97b5d8632ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:11.631888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e01b8f-4ea7-4dc7-a4c2-f97b5d8632ea None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:11.632391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e01b8f-4ea7-4dc7-a4c2-f97b5d8632ea None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:11.632864 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e01b8f-4ea7-4dc7-a4c2-f97b5d8632ea None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:11.633896 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e01b8f-4ea7-4dc7-a4c2-f97b5d8632ea None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:11.634414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e01b8f-4ea7-4dc7-a4c2-f97b5d8632ea None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:11.636325 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e01b8f-4ea7-4dc7-a4c2-f97b5d8632ea None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:11.636988 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ab352d-cc23-41c8-a18e-1b1bb595036e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:11.637726 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ab352d-cc23-41c8-a18e-1b1bb595036e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:11.638110 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ab352d-cc23-41c8-a18e-1b1bb595036e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:11.638491 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ab352d-cc23-41c8-a18e-1b1bb595036e None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:11.639550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ab352d-cc23-41c8-a18e-1b1bb595036e None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:11.639986 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ab352d-cc23-41c8-a18e-1b1bb595036e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:11.642100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ab352d-cc23-41c8-a18e-1b1bb595036e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:11.653625 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 465/941] 10.222.0.39 () {56 vars in 1126 bytes} [Fri Sep 30 09:28:11 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:11.653955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 477/942] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:28:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:11.660152 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ab352d-cc23-41c8-a18e-1b1bb595036e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:11.663770 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e01b8f-4ea7-4dc7-a4c2-f97b5d8632ea None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:11.695165 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:11.772249 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:11.777465 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ab352d-cc23-41c8-a18e-1b1bb595036e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:11.794959 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e01b8f-4ea7-4dc7-a4c2-f97b5d8632ea None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:11.816618 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8c1e30-2479-487b-91c3-b6fd8f42edce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:11.817654 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8c1e30-2479-487b-91c3-b6fd8f42edce None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:11.818140 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8c1e30-2479-487b-91c3-b6fd8f42edce None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:11.818692 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8c1e30-2479-487b-91c3-b6fd8f42edce None admin] PATH_INFO: `/v3/endpoints` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:11.819594 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8c1e30-2479-487b-91c3-b6fd8f42edce None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:11.821163 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8c1e30-2479-487b-91c3-b6fd8f42edce None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:11.836853 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b234991-c0aa-4250-a4ec-cc800f6e36b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:11.837816 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b234991-c0aa-4250-a4ec-cc800f6e36b5 None admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:11.838914 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b234991-c0aa-4250-a4ec-cc800f6e36b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:11.839383 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b234991-c0aa-4250-a4ec-cc800f6e36b5 None admin] PATH_INFO: `/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/7d6e0545a40a436d8c63820b795551a9/roles/fa71d6ff1b04452c98c5830f178feb1b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:11.841844 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b234991-c0aa-4250-a4ec-cc800f6e36b5 None admin] RBAC: Authorizing `identity:create_grant(project_id=c3802ced1fce4997a016d41eca843ef8, user_id=7d6e0545a40a436d8c63820b795551a9, role_id=fa71d6ff1b04452c98c5830f178feb1b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:11.843560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b234991-c0aa-4250-a4ec-cc800f6e36b5 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:11.934623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 466/943] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:28:11 2022] POST /identity/v3/endpoints => generated 347 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:28:11.961160 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 478/944] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:28:11 2022] PUT /identity/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/7d6e0545a40a436d8c63820b795551a9/roles/fa71d6ff1b04452c98c5830f178feb1b => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:28:12.387444 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871f2799-95df-4c16-b29e-8a83ffacec4e None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:12.387841 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871f2799-95df-4c16-b29e-8a83ffacec4e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:12.388032 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871f2799-95df-4c16-b29e-8a83ffacec4e None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:12.389542 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 467/945] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:12.394999 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74b6bec-f3fd-4f18-9fbb-555f45eb0795 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:12.395435 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74b6bec-f3fd-4f18-9fbb-555f45eb0795 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:12.395633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74b6bec-f3fd-4f18-9fbb-555f45eb0795 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:12.413950 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-b74b6bec-f3fd-4f18-9fbb-555f45eb0795 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:12.415178 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b74b6bec-f3fd-4f18-9fbb-555f45eb0795 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:12.494937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 479/946] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:12 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:12.761263 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871f2799-95df-4c16-b29e-8a83ffacec4e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:12.874057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871f2799-95df-4c16-b29e-8a83ffacec4e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:12.912773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51795e03-73a7-4bfc-9160-75db00c8a60e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:12.913401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51795e03-73a7-4bfc-9160-75db00c8a60e None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:12.913685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51795e03-73a7-4bfc-9160-75db00c8a60e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:12.913953 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51795e03-73a7-4bfc-9160-75db00c8a60e None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:12.914653 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51795e03-73a7-4bfc-9160-75db00c8a60e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:12.916652 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51795e03-73a7-4bfc-9160-75db00c8a60e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:12.919324 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: WARNING keystone.server.flask.common [None req-51795e03-73a7-4bfc-9160-75db00c8a60e None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 30 09:28:12.953305 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed4e968-df27-4337-a7c4-88851167b7ac None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:12.956935 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed4e968-df27-4337-a7c4-88851167b7ac None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:12.956935 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed4e968-df27-4337-a7c4-88851167b7ac None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:12.956935 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 480/947] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:12.960519 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c710fa99-873e-4cf4-8a4a-e2bd1660cfe7 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:12.960876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c710fa99-873e-4cf4-8a4a-e2bd1660cfe7 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:12.961218 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c710fa99-873e-4cf4-8a4a-e2bd1660cfe7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:12.976616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.sql.core [None req-51795e03-73a7-4bfc-9160-75db00c8a60e None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 30 09:28:12.976616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 30 09:28:12.976616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [parameters: {'id': 'd165071319044084ab8301e13d10179b', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 30 09:28:12.976616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81684) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 30 09:28:12.980583 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-c710fa99-873e-4cf4-8a4a-e2bd1660cfe7 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:12.981834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c710fa99-873e-4cf4-8a4a-e2bd1660cfe7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:12.984028 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:12.996730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-51795e03-73a7-4bfc-9160-75db00c8a60e None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id d165071319044084ab8301e13d10179b.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id d165071319044084ab8301e13d10179b. Sep 30 09:28:12.996730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:12.996730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 30 09:28:12.996730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 30 09:28:12.996730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 30 09:28:12.996730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 30 09:28:12.996730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 30 09:28:12.996730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application result = self._query(query) Sep 30 09:28:12.996730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 30 09:28:12.996730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application conn.query(q) Sep 30 09:28:12.996730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 30 09:28:12.996730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 30 09:28:12.996730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 30 09:28:12.997576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application result.read() Sep 30 09:28:12.997576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 30 09:28:12.997576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 30 09:28:12.997576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 30 09:28:12.997576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 30 09:28:12.997576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 30 09:28:12.997576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 30 09:28:12.997576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 30 09:28:12.997576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 30 09:28:12.997576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 30 09:28:12.997576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:12.997576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 30 09:28:12.997576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:12.997576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:12.997576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 30 09:28:12.997576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 30 09:28:12.997576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 30 09:28:12.997576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 30 09:28:12.998572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 30 09:28:12.998572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application next(self.gen) Sep 30 09:28:12.998572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 30 09:28:12.998572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application yield resource Sep 30 09:28:12.998572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 30 09:28:12.998572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application next(self.gen) Sep 30 09:28:12.998572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 30 09:28:12.998572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self.session.rollback() Sep 30 09:28:12.998572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 30 09:28:12.998572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self.force_reraise() Sep 30 09:28:12.998572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 30 09:28:12.998572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise self.value Sep 30 09:28:12.998572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 30 09:28:12.998572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 30 09:28:12.998572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 30 09:28:12.998572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application session.commit() Sep 30 09:28:12.998572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 30 09:28:12.999515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 30 09:28:12.999515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 30 09:28:12.999515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._prepare_impl() Sep 30 09:28:12.999515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 30 09:28:12.999515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self.session.flush() Sep 30 09:28:12.999515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 30 09:28:12.999515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._flush(objects) Sep 30 09:28:12.999515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 30 09:28:12.999515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 30 09:28:12.999515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 30 09:28:12.999515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application compat.raise_( Sep 30 09:28:12.999515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 30 09:28:12.999515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception Sep 30 09:28:12.999515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 30 09:28:12.999515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application flush_context.execute() Sep 30 09:28:12.999515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 30 09:28:12.999515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rec.execute(self) Sep 30 09:28:13.000425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 30 09:28:13.000425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 30 09:28:13.000425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 30 09:28:13.000425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 30 09:28:13.000425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 30 09:28:13.000425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 30 09:28:13.000425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 30 09:28:13.000425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 30 09:28:13.000425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 30 09:28:13.000425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 30 09:28:13.000425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 30 09:28:13.000425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 30 09:28:13.000425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 30 09:28:13.000425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 30 09:28:13.000425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 30 09:28:13.001243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 30 09:28:13.001243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 30 09:28:13.001243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception Sep 30 09:28:13.001243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 30 09:28:13.001243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 30 09:28:13.001243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 30 09:28:13.001243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 30 09:28:13.001243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 30 09:28:13.001243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application result = self._query(query) Sep 30 09:28:13.001243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 30 09:28:13.001243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application conn.query(q) Sep 30 09:28:13.001243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 30 09:28:13.001243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 30 09:28:13.001243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 30 09:28:13.001243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application result.read() Sep 30 09:28:13.001243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 30 09:28:13.001243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 30 09:28:13.002124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 30 09:28:13.002124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 30 09:28:13.002124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 30 09:28:13.002124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 30 09:28:13.002124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 30 09:28:13.002124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 30 09:28:13.002124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 30 09:28:13.002124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 30 09:28:13.002124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [parameters: {'id': 'd165071319044084ab8301e13d10179b', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 30 09:28:13.002124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 30 09:28:13.002124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:13.002124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:28:13.002124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:13.002124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:13.002124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 30 09:28:13.003101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 30 09:28:13.003101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 30 09:28:13.003101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 30 09:28:13.003101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 30 09:28:13.003101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:13.003101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:28:13.003101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:13.003101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:13.003101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:13.003101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:13.003101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:13.003101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:13.003101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:13.003101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:13.003101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:13.003101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:13.003101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:13.004271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:13.004271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 30 09:28:13.004271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 30 09:28:13.004271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:13.004271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:13.004271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 30 09:28:13.004271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 30 09:28:13.004271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id d165071319044084ab8301e13d10179b. Sep 30 09:28:13.004271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:13.006804 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 468/948] 10.222.0.39 () {60 vars in 1140 bytes} [Fri Sep 30 09:28:12 2022] POST /identity/v3/projects => generated 262 bytes in 246 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:13.012734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51795e03-73a7-4bfc-9160-75db00c8a60e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:13.029401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:13.060399 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 481/949] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:12 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 30 09:28:13.126999 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51795e03-73a7-4bfc-9160-75db00c8a60e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:13.161647 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e521c0c0-1574-47e3-87cf-300cc785692a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:13.162287 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e521c0c0-1574-47e3-87cf-300cc785692a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:13.162567 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e521c0c0-1574-47e3-87cf-300cc785692a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:13.162867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e521c0c0-1574-47e3-87cf-300cc785692a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:13.172685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e521c0c0-1574-47e3-87cf-300cc785692a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:13.180387 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e521c0c0-1574-47e3-87cf-300cc785692a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:13.192452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-e521c0c0-1574-47e3-87cf-300cc785692a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:28:13.192452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:13.192452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:13.192452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:13.192452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:13.192452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:13.192452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:13.192452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:13.192452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:13.192452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:13.192452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:13.192452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:13.192452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:28:13.192452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:28:13.192452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:28:13.193373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:28:13.193373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:13.193373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:13.193373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:13.193373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:13.193373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:13.193373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:13.193373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:13.193373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:13.193373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:13.193373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:13.193373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:13.193373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:13.193373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:13.193373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:13.193373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:13.193373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:13.194291 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:28:13.194291 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:28:13.194291 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:28:13.194291 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:28:13.194291 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:28:13.194291 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:28:13.194291 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:28:13.194291 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:13.199111 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 469/950] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:28:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:13.204889 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c710fa99-873e-4cf4-8a4a-e2bd1660cfe7 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:13.315777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c710fa99-873e-4cf4-8a4a-e2bd1660cfe7 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:13.337123 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e521c0c0-1574-47e3-87cf-300cc785692a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:13.358115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93be722c-b4c2-4ea5-8e40-7ccfbcbd1ac4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:13.358759 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93be722c-b4c2-4ea5-8e40-7ccfbcbd1ac4 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:13.359043 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93be722c-b4c2-4ea5-8e40-7ccfbcbd1ac4 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:13.359368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93be722c-b4c2-4ea5-8e40-7ccfbcbd1ac4 None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:13.360035 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93be722c-b4c2-4ea5-8e40-7ccfbcbd1ac4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:13.360340 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93be722c-b4c2-4ea5-8e40-7ccfbcbd1ac4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:13.362291 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93be722c-b4c2-4ea5-8e40-7ccfbcbd1ac4 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:13.377072 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:13.389328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 482/951] 10.222.0.39 () {56 vars in 1112 bytes} [Fri Sep 30 09:28:13 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:13.426006 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:13.461257 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e521c0c0-1574-47e3-87cf-300cc785692a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:13.498542 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b88daf8-8a6a-47f3-bb65-6dc7fd06719c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:13.499468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b88daf8-8a6a-47f3-bb65-6dc7fd06719c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:13.499953 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b88daf8-8a6a-47f3-bb65-6dc7fd06719c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:13.500408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b88daf8-8a6a-47f3-bb65-6dc7fd06719c None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:13.501216 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b88daf8-8a6a-47f3-bb65-6dc7fd06719c None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:13.502775 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b88daf8-8a6a-47f3-bb65-6dc7fd06719c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:13.513612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-2b88daf8-8a6a-47f3-bb65-6dc7fd06719c None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 30 09:28:13.513612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:13.513612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:28:13.513612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:28:13.513612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:28:13.513612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:28:13.513612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:28:13.513612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:13.513612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 30 09:28:13.513612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:13.513612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:28:13.513612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:13.513612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:13.513612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:13.513612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:13.513612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:13.513612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:13.515075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:13.515075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:13.515075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:13.515075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:13.515075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:13.515075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:13.515075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:28:13.515075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:28:13.515075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:28:13.515075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:28:13.515075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:13.515075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:13.515075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:13.515075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:13.515075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:13.515075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:13.516199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:13.516199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:13.516199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:13.516199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:13.516199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:13.516199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:13.516199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:13.516199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:13.516199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:13.516199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:13.516199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:28:13.516199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:13.516199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 30 09:28:13.516199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:13.518763 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 470/952] 10.222.0.39 () {56 vars in 1126 bytes} [Fri Sep 30 09:28:13 2022] GET /identity/v3/services/object-store => generated 93 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:13.525154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93be722c-b4c2-4ea5-8e40-7ccfbcbd1ac4 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:13.637911 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93be722c-b4c2-4ea5-8e40-7ccfbcbd1ac4 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:13.673635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f36d391-b85b-430d-aaf1-5af6ef2fe291 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:13.674245 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f36d391-b85b-430d-aaf1-5af6ef2fe291 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:13.674526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f36d391-b85b-430d-aaf1-5af6ef2fe291 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:13.674831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f36d391-b85b-430d-aaf1-5af6ef2fe291 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:13.675458 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f36d391-b85b-430d-aaf1-5af6ef2fe291 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:13.675758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f36d391-b85b-430d-aaf1-5af6ef2fe291 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:13.677131 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f36d391-b85b-430d-aaf1-5af6ef2fe291 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:13.683164 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 483/953] 10.222.0.39 () {56 vars in 1122 bytes} [Fri Sep 30 09:28:13 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:13.689031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b88daf8-8a6a-47f3-bb65-6dc7fd06719c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:13.797285 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b88daf8-8a6a-47f3-bb65-6dc7fd06719c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:13.833322 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b11c0e-255e-4d5c-9e3c-7cf2bcd16bb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:13.834208 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b11c0e-255e-4d5c-9e3c-7cf2bcd16bb4 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:13.834657 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b11c0e-255e-4d5c-9e3c-7cf2bcd16bb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:13.835094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b11c0e-255e-4d5c-9e3c-7cf2bcd16bb4 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:13.835881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b11c0e-255e-4d5c-9e3c-7cf2bcd16bb4 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:13.836328 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b11c0e-255e-4d5c-9e3c-7cf2bcd16bb4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:13.838349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b11c0e-255e-4d5c-9e3c-7cf2bcd16bb4 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:13.846901 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:13.850874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 471/954] 10.222.0.39 () {56 vars in 1122 bytes} [Fri Sep 30 09:28:13 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:13.852605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f36d391-b85b-430d-aaf1-5af6ef2fe291 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:13.916846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:13.959614 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f36d391-b85b-430d-aaf1-5af6ef2fe291 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:13.994849 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fcee44-9c2e-46f0-b998-cf0c2d4949cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:13.995463 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fcee44-9c2e-46f0-b998-cf0c2d4949cd None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:13.995744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fcee44-9c2e-46f0-b998-cf0c2d4949cd None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:13.996017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fcee44-9c2e-46f0-b998-cf0c2d4949cd None admin] PATH_INFO: `/v3/endpoints` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:13.996683 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fcee44-9c2e-46f0-b998-cf0c2d4949cd None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:13.998135 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fcee44-9c2e-46f0-b998-cf0c2d4949cd None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:14.100921 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 484/955] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:28:13 2022] POST /identity/v3/endpoints => generated 349 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:28:14.189377 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b221f7-80c7-4678-9483-6bd740e29f9a None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:14.189890 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b221f7-80c7-4678-9483-6bd740e29f9a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:14.189890 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b221f7-80c7-4678-9483-6bd740e29f9a None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:14.191489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 472/956] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:14.200003 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc46ad21-044c-4f07-a9f4-17098bd7510e None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:14.200295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc46ad21-044c-4f07-a9f4-17098bd7510e None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:14.200568 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc46ad21-044c-4f07-a9f4-17098bd7510e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:14.221110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-dc46ad21-044c-4f07-a9f4-17098bd7510e None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:14.222344 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-dc46ad21-044c-4f07-a9f4-17098bd7510e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:14.258225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7a9ef7-c434-4311-8126-3b794967ec66 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:14.258564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7a9ef7-c434-4311-8126-3b794967ec66 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:14.258819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7a9ef7-c434-4311-8126-3b794967ec66 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:14.260500 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 473/957] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:14.266059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd86ee6-0da8-4bb8-942e-fed5ff9c9839 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:14.266376 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd86ee6-0da8-4bb8-942e-fed5ff9c9839 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:14.266621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd86ee6-0da8-4bb8-942e-fed5ff9c9839 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:14.285710 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-acd86ee6-0da8-4bb8-942e-fed5ff9c9839 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:14.287296 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-acd86ee6-0da8-4bb8-942e-fed5ff9c9839 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:14.311838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 485/958] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:14 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:14.366474 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 474/959] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:14 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 30 09:28:14.581965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc46ad21-044c-4f07-a9f4-17098bd7510e None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:14.629954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:14.636918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd86ee6-0da8-4bb8-942e-fed5ff9c9839 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:14.686260 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc46ad21-044c-4f07-a9f4-17098bd7510e None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:14.723242 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ff37fb-d2a7-43f0-990f-1801340e69c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:14.723887 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ff37fb-d2a7-43f0-990f-1801340e69c5 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:14.724167 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ff37fb-d2a7-43f0-990f-1801340e69c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:14.724442 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ff37fb-d2a7-43f0-990f-1801340e69c5 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:14.741080 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ff37fb-d2a7-43f0-990f-1801340e69c5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:28:14.747969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-70ff37fb-d2a7-43f0-990f-1801340e69c5 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 30 09:28:14.747969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:14.747969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:14.747969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:14.747969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:14.747969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:14.747969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:14.747969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:14.747969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:14.747969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:14.747969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:14.747969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:14.747969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:28:14.747969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:28:14.747969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:28:14.747969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:28:14.748967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:28:14.748967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:28:14.748967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:28:14.748967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:28:14.748967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:14.748967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:14.748967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:14.748967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:14.748967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:14.748967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:14.748967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:14.748967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:14.748967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:14.748967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:14.748967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:14.748967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:14.749831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:14.749831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:14.749831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:14.749831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:14.749831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:28:14.749831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:28:14.749831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:28:14.749831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:28:14.749831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:28:14.749831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:28:14.749831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 30 09:28:14.749831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:14.751418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 486/960] 10.222.0.39 () {56 vars in 1120 bytes} [Fri Sep 30 09:28:14 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:14.755842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ff37fb-d2a7-43f0-990f-1801340e69c5 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:14.760123 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd86ee6-0da8-4bb8-942e-fed5ff9c9839 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:14.792391 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:14.806030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61d6105-55c4-43a1-ae2a-848fefaa304d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:14.806985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61d6105-55c4-43a1-ae2a-848fefaa304d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:14.807269 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61d6105-55c4-43a1-ae2a-848fefaa304d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:14.807547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61d6105-55c4-43a1-ae2a-848fefaa304d None admin] PATH_INFO: `/v3/services/compute` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:14.808129 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61d6105-55c4-43a1-ae2a-848fefaa304d None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:14.809493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61d6105-55c4-43a1-ae2a-848fefaa304d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:14.820279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-d61d6105-55c4-43a1-ae2a-848fefaa304d None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 30 09:28:14.820279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:14.820279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:28:14.820279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:28:14.820279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:28:14.820279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:28:14.820279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:28:14.820279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:14.820279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 30 09:28:14.820279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:14.820279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:28:14.820279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:14.820279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:14.820279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:14.820279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:14.820279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:14.820279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:14.821307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:14.821307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:14.821307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:14.821307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:14.821307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:14.821307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:14.821307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:28:14.821307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:28:14.821307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:28:14.821307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:28:14.821307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:14.821307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:14.821307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:14.821307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:14.821307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:14.821307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:14.822902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:14.822902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:14.822902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:14.822902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:14.822902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:14.822902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:14.822902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:14.822902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:14.822902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:14.822902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:14.822902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:28:14.822902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:14.822902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 30 09:28:14.822902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:14.830752 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 475/961] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:28:14 2022] GET /identity/v3/services/compute => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:14.831867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61d6105-55c4-43a1-ae2a-848fefaa304d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:14.891865 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ff37fb-d2a7-43f0-990f-1801340e69c5 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:14.894685 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:14.936776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851a3422-6c4d-4196-a1b4-3380bf5ba925 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:14.937391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851a3422-6c4d-4196-a1b4-3380bf5ba925 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:14.937955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851a3422-6c4d-4196-a1b4-3380bf5ba925 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:14.938258 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851a3422-6c4d-4196-a1b4-3380bf5ba925 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:14.938958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851a3422-6c4d-4196-a1b4-3380bf5ba925 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:14.939400 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851a3422-6c4d-4196-a1b4-3380bf5ba925 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:14.940947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851a3422-6c4d-4196-a1b4-3380bf5ba925 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:14.941770 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61d6105-55c4-43a1-ae2a-848fefaa304d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:14.951497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 487/962] 10.222.0.39 () {56 vars in 1115 bytes} [Fri Sep 30 09:28:14 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:14.956736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851a3422-6c4d-4196-a1b4-3380bf5ba925 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:14.983289 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cd13d8-2181-4755-88a1-8abf906ee436 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:14.984314 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cd13d8-2181-4755-88a1-8abf906ee436 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:14.984805 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cd13d8-2181-4755-88a1-8abf906ee436 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:14.985279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cd13d8-2181-4755-88a1-8abf906ee436 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:14.986248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cd13d8-2181-4755-88a1-8abf906ee436 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:14.986747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cd13d8-2181-4755-88a1-8abf906ee436 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:14.988365 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cd13d8-2181-4755-88a1-8abf906ee436 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:14.995591 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 476/963] 10.222.0.39 () {56 vars in 1112 bytes} [Fri Sep 30 09:28:14 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:15.001378 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cd13d8-2181-4755-88a1-8abf906ee436 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:15.070330 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:15.076080 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851a3422-6c4d-4196-a1b4-3380bf5ba925 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:15.118080 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0903c7-6012-40f2-b568-6df79acfa1da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:15.119184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0903c7-6012-40f2-b568-6df79acfa1da None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:15.119738 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0903c7-6012-40f2-b568-6df79acfa1da None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:15.120017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0903c7-6012-40f2-b568-6df79acfa1da None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:15.124711 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cd13d8-2181-4755-88a1-8abf906ee436 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:15.133588 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0903c7-6012-40f2-b568-6df79acfa1da None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:15.135806 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0903c7-6012-40f2-b568-6df79acfa1da None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:15.152994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-9a0903c7-6012-40f2-b568-6df79acfa1da None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:15.152994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:15.152994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:15.152994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:15.152994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:15.152994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:15.152994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:15.152994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:15.152994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:15.152994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:15.152994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:15.152994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:15.152994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:28:15.152994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:28:15.152994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:28:15.152994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:28:15.153962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:15.153962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:15.153962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:15.153962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:15.153962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:15.153962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:15.153962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:15.153962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:15.153962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:15.153962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:15.153962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:15.153962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:15.153962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:15.153962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:15.153962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:15.153962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:15.153962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:28:15.154953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:28:15.154953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:15.154953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:15.158801 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 488/964] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:15.159645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0903c7-6012-40f2-b568-6df79acfa1da None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:15.170124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b61210c-635b-4b3a-823c-0cd4febff7ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:15.171284 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b61210c-635b-4b3a-823c-0cd4febff7ce None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:15.171572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b61210c-635b-4b3a-823c-0cd4febff7ce None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:15.171846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b61210c-635b-4b3a-823c-0cd4febff7ce None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:15.172884 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b61210c-635b-4b3a-823c-0cd4febff7ce None admin] RBAC: Adding query filter params (type=compute) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:15.173191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b61210c-635b-4b3a-823c-0cd4febff7ce None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:15.175508 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b61210c-635b-4b3a-823c-0cd4febff7ce None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:15.180648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:15.186885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 477/965] 10.222.0.39 () {56 vars in 1112 bytes} [Fri Sep 30 09:28:15 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:15.282141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0903c7-6012-40f2-b568-6df79acfa1da None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:15.318876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e337ed7-519b-4a11-96e0-8ce1d94a1611 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:15.319502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e337ed7-519b-4a11-96e0-8ce1d94a1611 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:15.319780 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e337ed7-519b-4a11-96e0-8ce1d94a1611 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:15.320054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e337ed7-519b-4a11-96e0-8ce1d94a1611 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:15.320701 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e337ed7-519b-4a11-96e0-8ce1d94a1611 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:15.320999 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e337ed7-519b-4a11-96e0-8ce1d94a1611 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:15.323042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e337ed7-519b-4a11-96e0-8ce1d94a1611 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:15.336434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 489/966] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:28:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:15.342015 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b61210c-635b-4b3a-823c-0cd4febff7ce None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:15.448287 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b61210c-635b-4b3a-823c-0cd4febff7ce None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:15.468630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:15.483011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c91e5a2-4b37-45c1-8b49-2413a3d0b6dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:15.487118 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c91e5a2-4b37-45c1-8b49-2413a3d0b6dd None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:15.487118 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c91e5a2-4b37-45c1-8b49-2413a3d0b6dd None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:15.487118 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c91e5a2-4b37-45c1-8b49-2413a3d0b6dd None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:15.492216 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c91e5a2-4b37-45c1-8b49-2413a3d0b6dd None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:15.493667 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c91e5a2-4b37-45c1-8b49-2413a3d0b6dd None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:15.505881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-4c91e5a2-4b37-45c1-8b49-2413a3d0b6dd None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 30 09:28:15.505881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:15.505881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:15.505881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:15.505881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:15.505881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:15.505881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:15.505881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:15.505881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:15.505881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:15.505881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:15.505881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:15.505881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:28:15.505881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:28:15.505881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:28:15.505881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:28:15.506874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:15.506874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:15.506874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:28:15.506874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:28:15.506874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:28:15.506874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:28:15.506874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:15.506874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:15.506874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:15.506874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:15.506874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:15.506874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:15.506874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:15.506874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:15.506874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:15.506874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:15.506874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:15.506874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:15.507928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:15.507928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:15.507928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:28:15.507928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:28:15.507928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:28:15.507928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:28:15.507928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:28:15.507928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:28:15.507928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 30 09:28:15.507928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:15.507928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 478/967] 10.222.0.39 () {56 vars in 1117 bytes} [Fri Sep 30 09:28:15 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:15.513815 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e337ed7-519b-4a11-96e0-8ce1d94a1611 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:15.597815 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:15.620545 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e337ed7-519b-4a11-96e0-8ce1d94a1611 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:15.655808 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab64c59-3609-4533-903e-809e4ad64abf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:15.656421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab64c59-3609-4533-903e-809e4ad64abf None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:15.656706 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab64c59-3609-4533-903e-809e4ad64abf None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:15.656984 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab64c59-3609-4533-903e-809e4ad64abf None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:15.657712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab64c59-3609-4533-903e-809e4ad64abf None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:15.658022 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab64c59-3609-4533-903e-809e4ad64abf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:15.659502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab64c59-3609-4533-903e-809e4ad64abf None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:15.677910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 490/968] 10.222.0.39 () {56 vars in 1149 bytes} [Fri Sep 30 09:28:15 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:15.683895 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c91e5a2-4b37-45c1-8b49-2413a3d0b6dd None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:15.792544 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c91e5a2-4b37-45c1-8b49-2413a3d0b6dd None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:15.827661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a5ed2a-6fda-4922-9ae2-85684417d9a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:15.828271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a5ed2a-6fda-4922-9ae2-85684417d9a6 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:15.828552 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a5ed2a-6fda-4922-9ae2-85684417d9a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:15.828829 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a5ed2a-6fda-4922-9ae2-85684417d9a6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:15.841580 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec89488b-44a0-4a93-ab31-e9678439cbb6 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:15.842271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec89488b-44a0-4a93-ab31-e9678439cbb6 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:15.842815 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec89488b-44a0-4a93-ab31-e9678439cbb6 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:15.844634 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 491/969] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:15.849705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2e50d9-d0d3-4481-8c4c-642ffc7ac818 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:15.850208 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2e50d9-d0d3-4481-8c4c-642ffc7ac818 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:15.850716 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2e50d9-d0d3-4481-8c4c-642ffc7ac818 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:15.870526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-1c2e50d9-d0d3-4481-8c4c-642ffc7ac818 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:15.872064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-1c2e50d9-d0d3-4481-8c4c-642ffc7ac818 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:15.899003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:15.910269 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a5ed2a-6fda-4922-9ae2-85684417d9a6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:15.911939 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a5ed2a-6fda-4922-9ae2-85684417d9a6 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:15.950957 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 492/970] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:15 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:16.021470 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 479/971] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:28:15 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 30 09:28:16.027352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2e50d9-d0d3-4481-8c4c-642ffc7ac818 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:16.092881 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:16.132633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2e50d9-d0d3-4481-8c4c-642ffc7ac818 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:16.172019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4c1b7f-3513-4c66-953a-52e8e69e582a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:16.172657 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4c1b7f-3513-4c66-953a-52e8e69e582a None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:16.172940 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4c1b7f-3513-4c66-953a-52e8e69e582a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:16.173274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4c1b7f-3513-4c66-953a-52e8e69e582a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:16.184082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4c1b7f-3513-4c66-953a-52e8e69e582a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:16.185644 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4c1b7f-3513-4c66-953a-52e8e69e582a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:16.200797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-6b4c1b7f-3513-4c66-953a-52e8e69e582a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:16.200797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:16.200797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:16.200797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:16.200797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:16.200797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:16.200797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:16.200797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:16.200797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:16.200797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:16.200797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:16.200797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:16.200797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:28:16.200797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:28:16.200797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:28:16.200797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:28:16.201746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:16.201746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:16.201746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:16.201746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:16.201746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:16.201746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:16.201746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:16.201746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:16.201746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:16.201746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:16.201746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:16.201746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:16.201746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:16.201746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:16.201746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:16.201746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:16.201746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:28:16.202699 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:28:16.202699 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:16.202699 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:16.202699 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 493/972] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:16.207739 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a5ed2a-6fda-4922-9ae2-85684417d9a6 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:16.234083 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4c1b7f-3513-4c66-953a-52e8e69e582a None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:16.297880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:16.317598 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a5ed2a-6fda-4922-9ae2-85684417d9a6 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:16.361188 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d39fade-0448-4b85-b4fb-0d8c8167b73e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:16.361815 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d39fade-0448-4b85-b4fb-0d8c8167b73e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:16.362050 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d39fade-0448-4b85-b4fb-0d8c8167b73e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:16.362150 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4c1b7f-3513-4c66-953a-52e8e69e582a None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:16.362246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d39fade-0448-4b85-b4fb-0d8c8167b73e None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:16.362938 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d39fade-0448-4b85-b4fb-0d8c8167b73e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:16.363247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d39fade-0448-4b85-b4fb-0d8c8167b73e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:16.364994 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d39fade-0448-4b85-b4fb-0d8c8167b73e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:16.378255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 480/973] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:28:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:16.384002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d39fade-0448-4b85-b4fb-0d8c8167b73e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:16.387793 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:16.398853 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2d65f3-1531-4f85-b574-63883b0a8ea7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:16.399475 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2d65f3-1531-4f85-b574-63883b0a8ea7 None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:16.399758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2d65f3-1531-4f85-b574-63883b0a8ea7 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:16.402343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2d65f3-1531-4f85-b574-63883b0a8ea7 None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:16.402343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2d65f3-1531-4f85-b574-63883b0a8ea7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:16.402790 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2d65f3-1531-4f85-b574-63883b0a8ea7 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:16.405346 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: WARNING keystone.server.flask.common [None req-1e2d65f3-1531-4f85-b574-63883b0a8ea7 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 30 09:28:16.448337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.sql.core [None req-1e2d65f3-1531-4f85-b574-63883b0a8ea7 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 30 09:28:16.448337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 30 09:28:16.448337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [parameters: {'user_id': 'b8706f3e0140455a93f92c4e70089dcf', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 30 09:28:16.448337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81685) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 30 09:28:16.457744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-1e2d65f3-1531-4f85-b574-63883b0a8ea7 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 30 09:28:16.457744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:16.457744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 30 09:28:16.457744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 30 09:28:16.457744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 30 09:28:16.457744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 30 09:28:16.457744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 30 09:28:16.457744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application result = self._query(query) Sep 30 09:28:16.457744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 30 09:28:16.457744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application conn.query(q) Sep 30 09:28:16.457744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 30 09:28:16.457744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 30 09:28:16.457744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 30 09:28:16.457744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application result.read() Sep 30 09:28:16.457744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 30 09:28:16.458703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 30 09:28:16.458703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 30 09:28:16.458703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 30 09:28:16.458703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 30 09:28:16.458703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 30 09:28:16.458703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 30 09:28:16.458703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 30 09:28:16.458703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 30 09:28:16.458703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:16.458703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 30 09:28:16.458703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:16.458703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:16.458703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 30 09:28:16.458703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 30 09:28:16.458703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 30 09:28:16.458703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 30 09:28:16.458703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 30 09:28:16.458703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application next(self.gen) Sep 30 09:28:16.460280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 30 09:28:16.460280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application yield resource Sep 30 09:28:16.460280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 30 09:28:16.460280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application next(self.gen) Sep 30 09:28:16.460280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 30 09:28:16.460280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self.session.rollback() Sep 30 09:28:16.460280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 30 09:28:16.460280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self.force_reraise() Sep 30 09:28:16.460280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 30 09:28:16.460280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise self.value Sep 30 09:28:16.460280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 30 09:28:16.460280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 30 09:28:16.460280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 30 09:28:16.460280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application session.commit() Sep 30 09:28:16.460280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 30 09:28:16.460280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 30 09:28:16.460280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 30 09:28:16.461239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._prepare_impl() Sep 30 09:28:16.461239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 30 09:28:16.461239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self.session.flush() Sep 30 09:28:16.461239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 30 09:28:16.461239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._flush(objects) Sep 30 09:28:16.461239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 30 09:28:16.461239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 30 09:28:16.461239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 30 09:28:16.461239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application compat.raise_( Sep 30 09:28:16.461239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 30 09:28:16.461239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception Sep 30 09:28:16.461239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 30 09:28:16.461239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application flush_context.execute() Sep 30 09:28:16.461239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 30 09:28:16.461239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rec.execute(self) Sep 30 09:28:16.461239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 30 09:28:16.461239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 30 09:28:16.462209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 30 09:28:16.462209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 30 09:28:16.462209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 30 09:28:16.462209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 30 09:28:16.462209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 30 09:28:16.462209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 30 09:28:16.462209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 30 09:28:16.462209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 30 09:28:16.462209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 30 09:28:16.462209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 30 09:28:16.462209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 30 09:28:16.462209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 30 09:28:16.462209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 30 09:28:16.462209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 30 09:28:16.462209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 30 09:28:16.462209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception Sep 30 09:28:16.463715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 30 09:28:16.463715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 30 09:28:16.463715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 30 09:28:16.463715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 30 09:28:16.463715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 30 09:28:16.463715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application result = self._query(query) Sep 30 09:28:16.463715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 30 09:28:16.463715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application conn.query(q) Sep 30 09:28:16.463715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 30 09:28:16.463715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 30 09:28:16.463715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 30 09:28:16.463715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application result.read() Sep 30 09:28:16.463715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 30 09:28:16.463715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 30 09:28:16.463715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 30 09:28:16.463715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 30 09:28:16.470855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 30 09:28:16.470855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 30 09:28:16.470855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 30 09:28:16.470855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 30 09:28:16.470855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 30 09:28:16.470855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 30 09:28:16.470855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [parameters: {'user_id': 'b8706f3e0140455a93f92c4e70089dcf', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 30 09:28:16.470855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 30 09:28:16.470855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:16.470855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:28:16.470855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:16.470855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:16.470855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:16.470855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:16.470855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:16.471769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:16.471769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:16.471769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:16.471769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:16.471769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:16.471769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:16.471769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:16.471769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 30 09:28:16.471769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 30 09:28:16.471769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:16.471769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:16.471769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:28:16.471769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:28:16.471769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:28:16.471769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:28:16.471769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 30 09:28:16.471769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 30 09:28:16.472822 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 30 09:28:16.472822 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 30 09:28:16.472822 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 30 09:28:16.472822 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 30 09:28:16.472822 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 30 09:28:16.472822 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:16.472822 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 494/974] 10.222.0.39 () {60 vars in 1131 bytes} [Fri Sep 30 09:28:16 2022] POST /identity/v3/users => generated 160 bytes in 225 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:16.472822 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43237e3-5271-4c71-ad84-615f727bf684 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:16.472822 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43237e3-5271-4c71-ad84-615f727bf684 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:16.472822 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43237e3-5271-4c71-ad84-615f727bf684 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:16.473537 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 495/975] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:16 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:16.473537 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43237e3-5271-4c71-ad84-615f727bf684 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:16.516373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d39fade-0448-4b85-b4fb-0d8c8167b73e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:16.563371 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0bbc40-3bed-41b7-82db-743ec7bdbe9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:16.564015 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0bbc40-3bed-41b7-82db-743ec7bdbe9e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:16.564293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0bbc40-3bed-41b7-82db-743ec7bdbe9e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:16.564567 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0bbc40-3bed-41b7-82db-743ec7bdbe9e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:16.573645 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0bbc40-3bed-41b7-82db-743ec7bdbe9e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:16.575167 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0bbc40-3bed-41b7-82db-743ec7bdbe9e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:16.578185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:16.590104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-8b0bbc40-3bed-41b7-82db-743ec7bdbe9e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:28:16.590104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:16.590104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:16.590104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:16.590104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:16.590104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:16.590104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:16.590104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:16.590104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:16.590104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:16.590104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:16.590104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:16.590104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:28:16.590104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:28:16.590104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:28:16.591173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:28:16.591173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:16.591173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:16.591173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:16.591173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:16.591173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:16.591173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:16.591173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:16.591173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:16.591173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:16.591173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:16.591173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:16.591173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:16.591173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:16.591173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:16.591173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:16.591173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:16.592111 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:28:16.592111 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:28:16.592111 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:28:16.592111 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:28:16.592111 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:28:16.592111 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:28:16.592111 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:28:16.592111 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:16.596412 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 481/976] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:28:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:16.597978 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e6e057-b002-49d1-be2f-ef09bd2f92c1 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:16.598271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e6e057-b002-49d1-be2f-ef09bd2f92c1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:16.598551 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e6e057-b002-49d1-be2f-ef09bd2f92c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:16.598674 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43237e3-5271-4c71-ad84-615f727bf684 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:16.620451 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-45e6e057-b002-49d1-be2f-ef09bd2f92c1 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:16.622011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-45e6e057-b002-49d1-be2f-ef09bd2f92c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:16.637773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30838f1c-833e-43c6-8ceb-067d391e89d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:16.638679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30838f1c-833e-43c6-8ceb-067d391e89d5 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:16.639078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30838f1c-833e-43c6-8ceb-067d391e89d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:16.639428 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30838f1c-833e-43c6-8ceb-067d391e89d5 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:16.647512 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30838f1c-833e-43c6-8ceb-067d391e89d5 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:16.649620 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30838f1c-833e-43c6-8ceb-067d391e89d5 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:16.666838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-30838f1c-833e-43c6-8ceb-067d391e89d5 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 30 09:28:16.666838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:16.666838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:16.666838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:16.666838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:16.666838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:16.666838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:16.666838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:16.666838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:16.666838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:16.666838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:16.666838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:16.666838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:28:16.666838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:28:16.666838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:28:16.666838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:28:16.667876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:16.667876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:16.667876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:28:16.667876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:28:16.667876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:28:16.667876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:28:16.667876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:16.667876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:16.667876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:16.667876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:16.667876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:16.667876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:16.667876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:16.667876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:16.667876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:16.667876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:16.667876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:16.667876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:16.668890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:16.668890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:16.668890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:28:16.668890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:28:16.668890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:28:16.668890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:28:16.668890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:28:16.668890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:28:16.668890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 30 09:28:16.668890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:16.670743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 496/977] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:28:16 2022] GET /identity/v3/users/cinder => generated 84 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:16.671558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30838f1c-833e-43c6-8ceb-067d391e89d5 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:16.702155 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 482/978] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:16 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:16.705086 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e6e057-b002-49d1-be2f-ef09bd2f92c1 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:16.740074 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:16.846663 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30838f1c-833e-43c6-8ceb-067d391e89d5 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:16.863706 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e6e057-b002-49d1-be2f-ef09bd2f92c1 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:16.912997 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eedeab2a-5ca6-48f3-8a60-57117fd50353 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:16.914306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedeab2a-5ca6-48f3-8a60-57117fd50353 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:16.914607 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedeab2a-5ca6-48f3-8a60-57117fd50353 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:16.915047 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedeab2a-5ca6-48f3-8a60-57117fd50353 None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:16.915773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eedeab2a-5ca6-48f3-8a60-57117fd50353 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:16.916085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eedeab2a-5ca6-48f3-8a60-57117fd50353 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:16.918810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eedeab2a-5ca6-48f3-8a60-57117fd50353 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:16.923731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:16.931983 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17fce144-9b05-4626-98f9-694d1ec488cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:16.933451 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17fce144-9b05-4626-98f9-694d1ec488cf None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:16.933948 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17fce144-9b05-4626-98f9-694d1ec488cf None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:16.934722 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17fce144-9b05-4626-98f9-694d1ec488cf None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:16.935634 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17fce144-9b05-4626-98f9-694d1ec488cf None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:16.936185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17fce144-9b05-4626-98f9-694d1ec488cf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:16.937975 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17fce144-9b05-4626-98f9-694d1ec488cf None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:16.941454 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 497/979] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:28:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:16.942335 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eedeab2a-5ca6-48f3-8a60-57117fd50353 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:16.980649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 483/980] 10.222.0.39 () {56 vars in 1101 bytes} [Fri Sep 30 09:28:16 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:16.991775 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17fce144-9b05-4626-98f9-694d1ec488cf None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:17.052206 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:17.057427 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eedeab2a-5ca6-48f3-8a60-57117fd50353 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:17.097920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2aa3731-1624-4c13-9b13-430926f28352 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:17.099469 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2aa3731-1624-4c13-9b13-430926f28352 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:17.100107 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2aa3731-1624-4c13-9b13-430926f28352 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:17.100664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2aa3731-1624-4c13-9b13-430926f28352 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:17.102035 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2aa3731-1624-4c13-9b13-430926f28352 None admin] RBAC: Adding query filter params (role.id=fa71d6ff1b04452c98c5830f178feb1b, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=7d6e0545a40a436d8c63820b795551a9) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:17.103641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2aa3731-1624-4c13-9b13-430926f28352 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:17.105922 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2aa3731-1624-4c13-9b13-430926f28352 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:17.115892 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 498/981] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:28:16 2022] GET /identity/v3/role_assignments?user.id=7d6e0545a40a436d8c63820b795551a9&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=fa71d6ff1b04452c98c5830f178feb1b => generated 608 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:17.143333 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17fce144-9b05-4626-98f9-694d1ec488cf None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:17.192308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53826f5-ed52-40fd-9769-dcc0fdf482ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:17.192930 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53826f5-ed52-40fd-9769-dcc0fdf482ea None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:17.193216 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53826f5-ed52-40fd-9769-dcc0fdf482ea None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:17.193496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53826f5-ed52-40fd-9769-dcc0fdf482ea None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:17.194634 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53826f5-ed52-40fd-9769-dcc0fdf482ea None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:17.197105 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53826f5-ed52-40fd-9769-dcc0fdf482ea None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:17.205489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-e53826f5-ed52-40fd-9769-dcc0fdf482ea None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 30 09:28:17.205489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:17.205489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:28:17.205489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:28:17.205489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:28:17.205489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:28:17.205489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:28:17.205489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:17.205489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 30 09:28:17.205489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:17.205489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:28:17.205489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:17.205489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:17.205489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:17.205489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:17.205489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:17.205489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:17.206946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:17.206946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:17.206946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:17.206946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:17.206946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:17.206946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:17.206946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:28:17.206946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:28:17.206946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:28:17.206946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:28:17.206946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:17.206946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:17.206946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:17.206946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:17.206946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:17.206946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:17.207837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:17.207837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:17.207837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:17.207837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:17.207837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:17.207837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:17.207837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:17.207837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:17.207837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:17.207837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:17.207837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:28:17.207837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:17.207837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 30 09:28:17.207837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:17.207837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 484/982] 10.222.0.39 () {56 vars in 1126 bytes} [Fri Sep 30 09:28:16 2022] GET /identity/v3/services/object-store => generated 93 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:17.214310 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2aa3731-1624-4c13-9b13-430926f28352 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:17.340562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2aa3731-1624-4c13-9b13-430926f28352 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:17.376604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78715428-36ab-449c-91c8-8bfa2bdd8a36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:17.377248 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78715428-36ab-449c-91c8-8bfa2bdd8a36 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:17.377532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78715428-36ab-449c-91c8-8bfa2bdd8a36 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:17.377809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78715428-36ab-449c-91c8-8bfa2bdd8a36 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:17.378463 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78715428-36ab-449c-91c8-8bfa2bdd8a36 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:17.378803 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78715428-36ab-449c-91c8-8bfa2bdd8a36 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:17.380491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78715428-36ab-449c-91c8-8bfa2bdd8a36 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:17.390734 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 499/983] 10.222.0.39 () {56 vars in 1122 bytes} [Fri Sep 30 09:28:17 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:17.392854 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53826f5-ed52-40fd-9769-dcc0fdf482ea None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:17.448831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:17.497196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53826f5-ed52-40fd-9769-dcc0fdf482ea None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:17.504985 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cb74de-ae7d-458f-a9f5-f4a47a049bdd None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:17.505527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cb74de-ae7d-458f-a9f5-f4a47a049bdd None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:17.505804 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cb74de-ae7d-458f-a9f5-f4a47a049bdd None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:17.507311 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 500/984] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:17.512081 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c78ece8-f5f9-4a6f-b0a8-cbc30c0738b9 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:17.512366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c78ece8-f5f9-4a6f-b0a8-cbc30c0738b9 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:17.512647 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c78ece8-f5f9-4a6f-b0a8-cbc30c0738b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:17.531170 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-0c78ece8-f5f9-4a6f-b0a8-cbc30c0738b9 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:17.531915 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da1a85c-040d-4fdc-93fc-e49317691cd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:17.532533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da1a85c-040d-4fdc-93fc-e49317691cd7 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:17.532737 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0c78ece8-f5f9-4a6f-b0a8-cbc30c0738b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:17.532861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da1a85c-040d-4fdc-93fc-e49317691cd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:17.533084 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da1a85c-040d-4fdc-93fc-e49317691cd7 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:17.533711 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da1a85c-040d-4fdc-93fc-e49317691cd7 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:17.534019 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da1a85c-040d-4fdc-93fc-e49317691cd7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:17.535431 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da1a85c-040d-4fdc-93fc-e49317691cd7 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:17.545756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 485/985] 10.222.0.39 () {56 vars in 1122 bytes} [Fri Sep 30 09:28:17 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:17.547086 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da1a85c-040d-4fdc-93fc-e49317691cd7 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:17.560430 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:17.612774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 501/986] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:17 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 30 09:28:17.653793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da1a85c-040d-4fdc-93fc-e49317691cd7 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:17.671140 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:17.689022 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b809bcc-a84b-49c0-9391-08a78e32f0c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:17.689618 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b809bcc-a84b-49c0-9391-08a78e32f0c9 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:17.689898 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b809bcc-a84b-49c0-9391-08a78e32f0c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:17.690173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b809bcc-a84b-49c0-9391-08a78e32f0c9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:17.690962 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b809bcc-a84b-49c0-9391-08a78e32f0c9 None admin] RBAC: Adding query filter params (interface=admin, service_id=ecac99253aaa43bfa5c89f8a9e3e5288, region_id=RegionOne) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:17.691321 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b809bcc-a84b-49c0-9391-08a78e32f0c9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:17.692700 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b809bcc-a84b-49c0-9391-08a78e32f0c9 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:17.698751 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 486/987] 10.222.0.39 () {56 vars in 1249 bytes} [Fri Sep 30 09:28:17 2022] GET /identity/v3/endpoints?service_id=ecac99253aaa43bfa5c89f8a9e3e5288&interface=admin®ion_id=RegionOne => generated 193 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:17.703461 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c78ece8-f5f9-4a6f-b0a8-cbc30c0738b9 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:17.803681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c78ece8-f5f9-4a6f-b0a8-cbc30c0738b9 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:17.838049 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e83e37b-2f1c-4b7e-b6ee-e5e20d1b906d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:17.838689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e83e37b-2f1c-4b7e-b6ee-e5e20d1b906d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:17.838978 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e83e37b-2f1c-4b7e-b6ee-e5e20d1b906d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:17.839255 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e83e37b-2f1c-4b7e-b6ee-e5e20d1b906d None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:17.839856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e83e37b-2f1c-4b7e-b6ee-e5e20d1b906d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:17.841235 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e83e37b-2f1c-4b7e-b6ee-e5e20d1b906d None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:17.847416 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 502/988] 10.222.0.39 () {56 vars in 1087 bytes} [Fri Sep 30 09:28:17 2022] GET /identity/v3/services => generated 1497 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:28:17.887807 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b809bcc-a84b-49c0-9391-08a78e32f0c9 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:17.995839 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b809bcc-a84b-49c0-9391-08a78e32f0c9 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:18.033701 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd3adc0-f0b1-43a3-8386-80922020eb8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:18.034460 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd3adc0-f0b1-43a3-8386-80922020eb8a None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:18.034865 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd3adc0-f0b1-43a3-8386-80922020eb8a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:18.035239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd3adc0-f0b1-43a3-8386-80922020eb8a None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:18.035990 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd3adc0-f0b1-43a3-8386-80922020eb8a None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:18.037496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd3adc0-f0b1-43a3-8386-80922020eb8a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:18.122962 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 487/989] 10.222.0.39 () {60 vars in 1141 bytes} [Fri Sep 30 09:28:17 2022] POST /identity/v3/services => generated 242 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:28:19.341121 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7bf0c8-0802-49a1-8edc-3ae86e34d9eb None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:19.341620 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7bf0c8-0802-49a1-8edc-3ae86e34d9eb None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:19.341727 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7bf0c8-0802-49a1-8edc-3ae86e34d9eb None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:19.343083 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 503/990] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:19.348143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abc75ed-a3a4-4a4d-8a25-b873b40b668e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:19.348533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abc75ed-a3a4-4a4d-8a25-b873b40b668e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:19.348689 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abc75ed-a3a4-4a4d-8a25-b873b40b668e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:19.366873 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-3abc75ed-a3a4-4a4d-8a25-b873b40b668e None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:19.368076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3abc75ed-a3a4-4a4d-8a25-b873b40b668e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:19.402879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:19.441328 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 488/991] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:19 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:19.708776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7bf0c8-0802-49a1-8edc-3ae86e34d9eb None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:19.732994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:19.811495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7bf0c8-0802-49a1-8edc-3ae86e34d9eb None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:19.845886 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caac7e2a-d38a-4431-812c-177982ae6cbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:19.846520 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac7e2a-d38a-4431-812c-177982ae6cbf None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:19.846823 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac7e2a-d38a-4431-812c-177982ae6cbf None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:19.847096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac7e2a-d38a-4431-812c-177982ae6cbf None admin] PATH_INFO: `/v3/services/image` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:19.847669 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caac7e2a-d38a-4431-812c-177982ae6cbf None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:19.849042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caac7e2a-d38a-4431-812c-177982ae6cbf None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:19.857946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-caac7e2a-d38a-4431-812c-177982ae6cbf None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 30 09:28:19.857946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:19.857946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:28:19.857946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:28:19.857946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:28:19.857946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:28:19.857946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:28:19.857946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:19.857946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 30 09:28:19.857946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:19.857946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:28:19.857946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:19.857946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:19.857946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:19.857946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:19.857946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:19.857946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:19.858986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:19.858986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:19.858986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:19.858986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:19.858986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:19.858986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:19.858986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:28:19.858986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:28:19.858986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:28:19.858986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:28:19.858986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:19.858986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:19.858986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:19.858986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:19.858986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:19.858986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:19.859845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:19.859845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:19.859845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:19.859845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:19.859845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:19.859845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:19.859845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:19.859845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:19.859845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:19.859845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:19.859845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:28:19.859845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:19.859845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 30 09:28:19.859845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:19.861364 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 504/992] 10.222.0.39 () {56 vars in 1105 bytes} [Fri Sep 30 09:28:19 2022] GET /identity/v3/services/image => generated 86 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:19.865110 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abc75ed-a3a4-4a4d-8a25-b873b40b668e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:19.969358 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abc75ed-a3a4-4a4d-8a25-b873b40b668e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:20.002552 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1706ee-4353-4b3a-8e3f-03e1f47f5509 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:20.003180 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1706ee-4353-4b3a-8e3f-03e1f47f5509 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:20.003460 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1706ee-4353-4b3a-8e3f-03e1f47f5509 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:20.003734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1706ee-4353-4b3a-8e3f-03e1f47f5509 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:20.004354 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1706ee-4353-4b3a-8e3f-03e1f47f5509 None admin] RBAC: Adding query filter params (name=image) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:20.004655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1706ee-4353-4b3a-8e3f-03e1f47f5509 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:20.006335 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1706ee-4353-4b3a-8e3f-03e1f47f5509 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:20.012474 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 489/993] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:19 2022] GET /identity/v3/services?name=image => generated 122 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:20.017427 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1706ee-4353-4b3a-8e3f-03e1f47f5509 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:20.018080 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1500a8-f105-412f-b56b-6cc2ff667dd1 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:20.018370 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1500a8-f105-412f-b56b-6cc2ff667dd1 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:20.018670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1500a8-f105-412f-b56b-6cc2ff667dd1 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:20.020867 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 505/994] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:20.025173 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0763483c-d297-4a64-b136-789de425910d None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:20.025455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0763483c-d297-4a64-b136-789de425910d None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:20.025894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0763483c-d297-4a64-b136-789de425910d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:20.044160 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-0763483c-d297-4a64-b136-789de425910d None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:20.048793 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0763483c-d297-4a64-b136-789de425910d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:20.049405 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:20.119707 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 506/995] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:20 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 30 09:28:20.125035 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1706ee-4353-4b3a-8e3f-03e1f47f5509 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:20.159735 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021557ea-a0f6-423a-b2c9-f282331a7197 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:20.160342 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021557ea-a0f6-423a-b2c9-f282331a7197 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:20.160623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021557ea-a0f6-423a-b2c9-f282331a7197 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:20.160897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021557ea-a0f6-423a-b2c9-f282331a7197 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:20.161569 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021557ea-a0f6-423a-b2c9-f282331a7197 None admin] RBAC: Adding query filter params (type=image) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:20.161874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021557ea-a0f6-423a-b2c9-f282331a7197 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:20.163338 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021557ea-a0f6-423a-b2c9-f282331a7197 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:20.174197 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 490/996] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:20 2022] GET /identity/v3/services?type=image => generated 122 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:20.296974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27c46df-0ab4-46aa-b6cf-08a9cd1c0996 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:20.297335 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27c46df-0ab4-46aa-b6cf-08a9cd1c0996 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:20.297524 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27c46df-0ab4-46aa-b6cf-08a9cd1c0996 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:20.299163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 507/997] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:20.304630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e4d93b-2c5a-4bc9-aec5-fc22732117bd None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:20.304919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e4d93b-2c5a-4bc9-aec5-fc22732117bd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:20.305194 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e4d93b-2c5a-4bc9-aec5-fc22732117bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:20.324800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d6e4d93b-2c5a-4bc9-aec5-fc22732117bd None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:20.326368 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d6e4d93b-2c5a-4bc9-aec5-fc22732117bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:20.390137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27c46df-0ab4-46aa-b6cf-08a9cd1c0996 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:20.398747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 491/998] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:20 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:20.409925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:20.495212 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27c46df-0ab4-46aa-b6cf-08a9cd1c0996 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:20.529051 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2d1ba2-bc35-4eae-9464-cbf2b399915d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:20.529681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2d1ba2-bc35-4eae-9464-cbf2b399915d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:20.529962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2d1ba2-bc35-4eae-9464-cbf2b399915d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:20.530234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2d1ba2-bc35-4eae-9464-cbf2b399915d None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:20.530833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2d1ba2-bc35-4eae-9464-cbf2b399915d None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:20.532209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2d1ba2-bc35-4eae-9464-cbf2b399915d None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:20.541641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-9a2d1ba2-bc35-4eae-9464-cbf2b399915d None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 30 09:28:20.541641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:20.541641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:28:20.541641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:28:20.541641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:28:20.541641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:28:20.541641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:28:20.541641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:20.541641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 30 09:28:20.541641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:20.541641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:28:20.541641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:20.541641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:20.541641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:20.541641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:20.541641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:20.541641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:20.542664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:20.542664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:20.542664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:20.542664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:20.542664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:20.542664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:20.542664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:28:20.542664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:28:20.542664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:28:20.542664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:28:20.542664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:20.542664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:20.542664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:20.542664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:20.542664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:20.542664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:20.543556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:20.543556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:20.543556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:20.543556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:20.543556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:20.543556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:20.543556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:20.543556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:20.543556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:20.543556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:20.543556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:28:20.543556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:20.543556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 30 09:28:20.543556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:20.546905 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 508/999] 10.222.0.39 () {56 vars in 1126 bytes} [Fri Sep 30 09:28:20 2022] GET /identity/v3/services/object-store => generated 93 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:20.548646 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e4d93b-2c5a-4bc9-aec5-fc22732117bd None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:20.573942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:20.648394 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e4d93b-2c5a-4bc9-aec5-fc22732117bd None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:20.668920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2d1ba2-bc35-4eae-9464-cbf2b399915d None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:20.684097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4283db98-cc2f-4dc4-87b2-abeb9ccc2f08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:20.684932 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4283db98-cc2f-4dc4-87b2-abeb9ccc2f08 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:20.685361 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4283db98-cc2f-4dc4-87b2-abeb9ccc2f08 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:20.685781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4283db98-cc2f-4dc4-87b2-abeb9ccc2f08 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:20.686550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4283db98-cc2f-4dc4-87b2-abeb9ccc2f08 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:20.687031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4283db98-cc2f-4dc4-87b2-abeb9ccc2f08 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:20.688553 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4283db98-cc2f-4dc4-87b2-abeb9ccc2f08 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:20.693564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 492/1000] 10.222.0.39 () {56 vars in 1122 bytes} [Fri Sep 30 09:28:20 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:20.699717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4283db98-cc2f-4dc4-87b2-abeb9ccc2f08 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:20.760770 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:20.781528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2d1ba2-bc35-4eae-9464-cbf2b399915d None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:20.796114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:20.812899 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4283db98-cc2f-4dc4-87b2-abeb9ccc2f08 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:20.818273 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9822e7e-e2d7-4870-866c-707f8e5356c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:20.818919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9822e7e-e2d7-4870-866c-707f8e5356c4 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:20.819307 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9822e7e-e2d7-4870-866c-707f8e5356c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:20.819587 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9822e7e-e2d7-4870-866c-707f8e5356c4 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:20.820190 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9822e7e-e2d7-4870-866c-707f8e5356c4 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:20.822071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9822e7e-e2d7-4870-866c-707f8e5356c4 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:20.831664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-e9822e7e-e2d7-4870-866c-707f8e5356c4 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 30 09:28:20.831664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:20.831664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:28:20.831664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:28:20.831664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:28:20.831664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:28:20.831664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:28:20.831664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:20.831664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 30 09:28:20.831664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:20.831664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:28:20.831664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:20.831664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:20.831664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:20.831664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:20.831664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:20.831664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:20.832741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:20.832741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:20.832741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:20.832741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:20.832741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:20.832741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:20.832741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:28:20.832741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:28:20.832741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:28:20.832741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:28:20.832741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:20.832741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:20.832741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:20.832741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:20.832741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:20.832741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:20.835802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:20.835802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:20.835802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:20.835802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:20.835802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:20.835802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:20.835802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:20.835802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:20.835802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:20.835802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:20.835802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:28:20.835802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:20.835802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 30 09:28:20.835802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:20.835802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 509/1001] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:28:20 2022] GET /identity/v3/services/compute => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:20.839126 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9822e7e-e2d7-4870-866c-707f8e5356c4 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:20.852457 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495d5ef4-1c9f-430f-8ad3-5ac3f20de525 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:20.853120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495d5ef4-1c9f-430f-8ad3-5ac3f20de525 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:20.853400 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495d5ef4-1c9f-430f-8ad3-5ac3f20de525 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:20.853670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495d5ef4-1c9f-430f-8ad3-5ac3f20de525 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:20.854455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495d5ef4-1c9f-430f-8ad3-5ac3f20de525 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:20.854794 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495d5ef4-1c9f-430f-8ad3-5ac3f20de525 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:20.856832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495d5ef4-1c9f-430f-8ad3-5ac3f20de525 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:20.862023 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 493/1002] 10.222.0.39 () {56 vars in 1122 bytes} [Fri Sep 30 09:28:20 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:20.867565 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495d5ef4-1c9f-430f-8ad3-5ac3f20de525 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:20.949239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9822e7e-e2d7-4870-866c-707f8e5356c4 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:20.970448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495d5ef4-1c9f-430f-8ad3-5ac3f20de525 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:20.986484 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de8eacb-353a-4adf-8e10-f7136fb7548f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:20.987119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de8eacb-353a-4adf-8e10-f7136fb7548f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:20.987401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de8eacb-353a-4adf-8e10-f7136fb7548f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:20.987682 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de8eacb-353a-4adf-8e10-f7136fb7548f None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:20.988311 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de8eacb-353a-4adf-8e10-f7136fb7548f None admin] RBAC: Adding query filter params (name=compute) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:20.988615 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de8eacb-353a-4adf-8e10-f7136fb7548f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:20.989993 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de8eacb-353a-4adf-8e10-f7136fb7548f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:20.995101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 510/1003] 10.222.0.39 () {56 vars in 1112 bytes} [Fri Sep 30 09:28:20 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:21.000845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de8eacb-353a-4adf-8e10-f7136fb7548f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:21.007593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4262fa-9312-48ea-bd58-e7e40f6f70d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:21.008206 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4262fa-9312-48ea-bd58-e7e40f6f70d6 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:21.008487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4262fa-9312-48ea-bd58-e7e40f6f70d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:21.008763 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4262fa-9312-48ea-bd58-e7e40f6f70d6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:21.009957 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4262fa-9312-48ea-bd58-e7e40f6f70d6 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:21.011357 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4262fa-9312-48ea-bd58-e7e40f6f70d6 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:21.012545 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:21.069275 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 494/1004] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:28:20 2022] POST /identity/v3/endpoints => generated 325 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:28:21.119814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de8eacb-353a-4adf-8e10-f7136fb7548f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:21.163994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a317f6b-6527-4e8b-a37b-674db8f70c20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:21.164609 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a317f6b-6527-4e8b-a37b-674db8f70c20 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:21.164885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a317f6b-6527-4e8b-a37b-674db8f70c20 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:21.165152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a317f6b-6527-4e8b-a37b-674db8f70c20 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:21.165824 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a317f6b-6527-4e8b-a37b-674db8f70c20 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:21.166120 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a317f6b-6527-4e8b-a37b-674db8f70c20 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:21.167537 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a317f6b-6527-4e8b-a37b-674db8f70c20 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:21.178166 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 511/1005] 10.222.0.39 () {56 vars in 1112 bytes} [Fri Sep 30 09:28:21 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:21.179135 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4262fa-9312-48ea-bd58-e7e40f6f70d6 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:21.199032 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:21.287166 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4262fa-9312-48ea-bd58-e7e40f6f70d6 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:21.326439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398c4591-c44a-49a7-88bc-9a87fc8c9db6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:21.327010 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398c4591-c44a-49a7-88bc-9a87fc8c9db6 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:21.327295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398c4591-c44a-49a7-88bc-9a87fc8c9db6 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:21.327579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398c4591-c44a-49a7-88bc-9a87fc8c9db6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:21.328273 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398c4591-c44a-49a7-88bc-9a87fc8c9db6 None admin] RBAC: Adding query filter params (interface=public, service_id=efa9a5f2604c436aa42728b593048943, region_id=RegionOne) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:21.328585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398c4591-c44a-49a7-88bc-9a87fc8c9db6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:21.329948 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398c4591-c44a-49a7-88bc-9a87fc8c9db6 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:21.335290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 495/1006] 10.222.0.39 () {56 vars in 1251 bytes} [Fri Sep 30 09:28:21 2022] GET /identity/v3/endpoints?service_id=efa9a5f2604c436aa42728b593048943&interface=public®ion_id=RegionOne => generated 194 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:21.340943 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a317f6b-6527-4e8b-a37b-674db8f70c20 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:21.411127 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:21.444492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a317f6b-6527-4e8b-a37b-674db8f70c20 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:21.479231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d534f710-16aa-4e07-8974-160e66e64977 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:21.479842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d534f710-16aa-4e07-8974-160e66e64977 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:21.480126 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d534f710-16aa-4e07-8974-160e66e64977 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:21.480398 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d534f710-16aa-4e07-8974-160e66e64977 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:21.480996 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d534f710-16aa-4e07-8974-160e66e64977 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:21.482371 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d534f710-16aa-4e07-8974-160e66e64977 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:21.488722 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 512/1007] 10.222.0.39 () {56 vars in 1087 bytes} [Fri Sep 30 09:28:21 2022] GET /identity/v3/services => generated 1727 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:28:22.342752 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bad751-8fa9-484a-95f9-3e8d333051c0 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:22.343258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bad751-8fa9-484a-95f9-3e8d333051c0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:22.343357 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bad751-8fa9-484a-95f9-3e8d333051c0 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:22.344776 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 496/1008] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:22.349839 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf2f6e1-3524-434f-84f9-44355b75d41d None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:22.350133 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf2f6e1-3524-434f-84f9-44355b75d41d None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:22.350406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf2f6e1-3524-434f-84f9-44355b75d41d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:22.369451 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-ecf2f6e1-3524-434f-84f9-44355b75d41d None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:22.370717 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ecf2f6e1-3524-434f-84f9-44355b75d41d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:22.439246 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 513/1009] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:22 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:22.704932 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bad751-8fa9-484a-95f9-3e8d333051c0 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:22.785599 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:22.809549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bad751-8fa9-484a-95f9-3e8d333051c0 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:22.844678 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5ee222-85c6-433a-a101-aa36a89f4a83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:22.845285 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5ee222-85c6-433a-a101-aa36a89f4a83 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:22.845560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5ee222-85c6-433a-a101-aa36a89f4a83 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:22.845829 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5ee222-85c6-433a-a101-aa36a89f4a83 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:22.846471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5ee222-85c6-433a-a101-aa36a89f4a83 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:22.847851 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5ee222-85c6-433a-a101-aa36a89f4a83 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:22.950384 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 497/1010] 10.222.0.39 () {60 vars in 1141 bytes} [Fri Sep 30 09:28:22 2022] POST /identity/v3/services => generated 242 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:28:23.256994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51aec5e-5642-4e26-bbdd-e08eb7da318a None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:23.257314 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51aec5e-5642-4e26-bbdd-e08eb7da318a None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:23.257542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51aec5e-5642-4e26-bbdd-e08eb7da318a None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:23.259609 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 514/1011] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:23.264595 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffee2bc-3ee3-406c-9d05-b1c53f23defb None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:23.264884 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffee2bc-3ee3-406c-9d05-b1c53f23defb None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:23.265161 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffee2bc-3ee3-406c-9d05-b1c53f23defb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:23.283809 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-0ffee2bc-3ee3-406c-9d05-b1c53f23defb None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:23.285002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0ffee2bc-3ee3-406c-9d05-b1c53f23defb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:23.359254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 498/1012] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:23 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:23.626174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51aec5e-5642-4e26-bbdd-e08eb7da318a None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:23.649206 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bea40b-8579-4f90-92af-9dc97002ad1d None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:23.649555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bea40b-8579-4f90-92af-9dc97002ad1d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:23.649817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bea40b-8579-4f90-92af-9dc97002ad1d None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:23.651218 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 499/1013] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:28:23.656751 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207b13b9-33fa-40ba-b63c-f5cf5fe0018d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:23.657016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207b13b9-33fa-40ba-b63c-f5cf5fe0018d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:23.657290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207b13b9-33fa-40ba-b63c-f5cf5fe0018d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:23.676670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-207b13b9-33fa-40ba-b63c-f5cf5fe0018d None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:23.677858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-207b13b9-33fa-40ba-b63c-f5cf5fe0018d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:23.692369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:23.733060 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51aec5e-5642-4e26-bbdd-e08eb7da318a None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:23.744261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:23.757677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 500/1014] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:23 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:23.768310 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe952d05-7c45-49ec-9694-3db8b8e37ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:23.768948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe952d05-7c45-49ec-9694-3db8b8e37ce7 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:23.769230 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe952d05-7c45-49ec-9694-3db8b8e37ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:23.769507 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe952d05-7c45-49ec-9694-3db8b8e37ce7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:23.770882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe952d05-7c45-49ec-9694-3db8b8e37ce7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:23.772343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe952d05-7c45-49ec-9694-3db8b8e37ce7 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:23.774638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 515/1015] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:23 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:23.780035 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207b13b9-33fa-40ba-b63c-f5cf5fe0018d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:23.879398 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207b13b9-33fa-40ba-b63c-f5cf5fe0018d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:23.912433 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bba3226-1d19-43bd-8dee-78e2d5f0d9ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:23.913032 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bba3226-1d19-43bd-8dee-78e2d5f0d9ef None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:23.913308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bba3226-1d19-43bd-8dee-78e2d5f0d9ef None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:23.913575 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bba3226-1d19-43bd-8dee-78e2d5f0d9ef None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:23.914204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bba3226-1d19-43bd-8dee-78e2d5f0d9ef None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:23.916171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bba3226-1d19-43bd-8dee-78e2d5f0d9ef None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:24.013835 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 501/1016] 10.222.0.39 () {60 vars in 1141 bytes} [Fri Sep 30 09:28:23 2022] POST /identity/v3/projects => generated 311 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:28:24.028797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe952d05-7c45-49ec-9694-3db8b8e37ce7 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:24.140458 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe952d05-7c45-49ec-9694-3db8b8e37ce7 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:24.175600 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:24.182357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67384b9-9425-4057-9841-38085b063e64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:24.183042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67384b9-9425-4057-9841-38085b063e64 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:24.186134 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67384b9-9425-4057-9841-38085b063e64 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:24.186134 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67384b9-9425-4057-9841-38085b063e64 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:24.186134 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67384b9-9425-4057-9841-38085b063e64 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:24.186134 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67384b9-9425-4057-9841-38085b063e64 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:24.195491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-d67384b9-9425-4057-9841-38085b063e64 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 30 09:28:24.195491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:24.195491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:28:24.195491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:28:24.195491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:28:24.195491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:28:24.195491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:28:24.195491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:24.195491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 30 09:28:24.195491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:24.195491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:28:24.195491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:24.195491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:24.195491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:24.195491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:24.195491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:24.195491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:24.196665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:24.196665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:24.196665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:24.196665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:24.196665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:24.196665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:24.196665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:28:24.196665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:28:24.196665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:28:24.196665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:28:24.196665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:24.196665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:24.196665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:24.196665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:24.196665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:24.196665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:24.197526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:24.197526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:24.197526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:24.197526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:24.197526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:24.197526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:24.197526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:24.197526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:24.197526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:24.197526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:24.197526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:28:24.197526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:24.197526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 30 09:28:24.197526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:24.201676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 516/1017] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:28:24 2022] GET /identity/v3/services/compute => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:24.202760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bba3226-1d19-43bd-8dee-78e2d5f0d9ef None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:24.253166 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:24.309053 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bba3226-1d19-43bd-8dee-78e2d5f0d9ef None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:24.342312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acdbc32-4f1f-434b-9818-ff3b339d7960 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:24.342945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acdbc32-4f1f-434b-9818-ff3b339d7960 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:24.343221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acdbc32-4f1f-434b-9818-ff3b339d7960 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:24.343487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acdbc32-4f1f-434b-9818-ff3b339d7960 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:24.344098 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7acdbc32-4f1f-434b-9818-ff3b339d7960 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:24.344396 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7acdbc32-4f1f-434b-9818-ff3b339d7960 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:24.345742 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7acdbc32-4f1f-434b-9818-ff3b339d7960 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:24.350488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 502/1018] 10.222.0.39 () {56 vars in 1112 bytes} [Fri Sep 30 09:28:24 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:24.356670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67384b9-9425-4057-9841-38085b063e64 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:24.456624 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67384b9-9425-4057-9841-38085b063e64 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:24.491645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99410657-2260-43f6-bb80-b5a0a3c1fdc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:24.492248 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99410657-2260-43f6-bb80-b5a0a3c1fdc8 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:24.492525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99410657-2260-43f6-bb80-b5a0a3c1fdc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:24.492795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99410657-2260-43f6-bb80-b5a0a3c1fdc8 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:24.493437 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99410657-2260-43f6-bb80-b5a0a3c1fdc8 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:24.493735 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99410657-2260-43f6-bb80-b5a0a3c1fdc8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:24.495123 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99410657-2260-43f6-bb80-b5a0a3c1fdc8 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:24.504531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 517/1019] 10.222.0.39 () {56 vars in 1112 bytes} [Fri Sep 30 09:28:24 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:24.506274 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acdbc32-4f1f-434b-9818-ff3b339d7960 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:24.607977 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acdbc32-4f1f-434b-9818-ff3b339d7960 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:24.616757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:24.641541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e048c95-92e8-4387-9dc9-0c786674dce1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:24.642145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e048c95-92e8-4387-9dc9-0c786674dce1 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:24.642422 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e048c95-92e8-4387-9dc9-0c786674dce1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:24.642721 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e048c95-92e8-4387-9dc9-0c786674dce1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:24.643382 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e048c95-92e8-4387-9dc9-0c786674dce1 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:24.644734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e048c95-92e8-4387-9dc9-0c786674dce1 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:24.726563 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 503/1020] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:28:24 2022] POST /identity/v3/endpoints => generated 334 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:28:25.139006 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ae0d45-26d5-4c87-918b-f8bbdc2fc127 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:25.139511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ae0d45-26d5-4c87-918b-f8bbdc2fc127 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:25.139511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ae0d45-26d5-4c87-918b-f8bbdc2fc127 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:25.140951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 518/1021] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:25.145959 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917cfd58-08fe-4967-aafe-cee49d07052b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:25.146252 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917cfd58-08fe-4967-aafe-cee49d07052b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:25.146523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917cfd58-08fe-4967-aafe-cee49d07052b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:25.164453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-917cfd58-08fe-4967-aafe-cee49d07052b None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:25.165683 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-917cfd58-08fe-4967-aafe-cee49d07052b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:25.245375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 504/1022] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:25 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:25.511536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ae0d45-26d5-4c87-918b-f8bbdc2fc127 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:25.572682 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:25.615064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ae0d45-26d5-4c87-918b-f8bbdc2fc127 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:25.648973 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e344159c-15b8-49cb-952e-76db3967db18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:25.650161 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e344159c-15b8-49cb-952e-76db3967db18 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:25.650434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e344159c-15b8-49cb-952e-76db3967db18 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:25.650730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e344159c-15b8-49cb-952e-76db3967db18 None admin] PATH_INFO: `/v3/services/image` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:25.651295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e344159c-15b8-49cb-952e-76db3967db18 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:25.652657 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e344159c-15b8-49cb-952e-76db3967db18 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:25.661496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-e344159c-15b8-49cb-952e-76db3967db18 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 30 09:28:25.661496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:25.661496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:28:25.661496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:28:25.661496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:28:25.661496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:28:25.661496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:28:25.661496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:25.661496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 30 09:28:25.661496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:25.661496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:28:25.661496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:25.661496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:25.661496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:25.661496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:25.661496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:25.661496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:25.662514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:25.662514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:25.662514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:25.662514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:25.662514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:25.662514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:25.662514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:28:25.662514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:28:25.662514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:28:25.662514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:28:25.662514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:25.662514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:25.662514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:25.662514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:25.662514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:25.662514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:25.663739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:25.663739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:25.663739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:25.663739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:25.663739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:25.663739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:25.663739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:25.663739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:25.663739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:25.663739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:25.663739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:28:25.663739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:25.663739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 30 09:28:25.663739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:25.666920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 519/1023] 10.222.0.39 () {56 vars in 1105 bytes} [Fri Sep 30 09:28:25 2022] GET /identity/v3/services/image => generated 86 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:25.668451 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917cfd58-08fe-4967-aafe-cee49d07052b None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:25.765520 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:25.775255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917cfd58-08fe-4967-aafe-cee49d07052b None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:25.809558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307d3e7a-d476-43c9-b1e9-8bab0108eae3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:25.810168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307d3e7a-d476-43c9-b1e9-8bab0108eae3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:25.810547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307d3e7a-d476-43c9-b1e9-8bab0108eae3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:25.810852 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307d3e7a-d476-43c9-b1e9-8bab0108eae3 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:25.811470 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307d3e7a-d476-43c9-b1e9-8bab0108eae3 None admin] RBAC: Adding query filter params (name=image) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:25.811771 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307d3e7a-d476-43c9-b1e9-8bab0108eae3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:25.813126 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307d3e7a-d476-43c9-b1e9-8bab0108eae3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:25.823072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 505/1024] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:25 2022] GET /identity/v3/services?name=image => generated 122 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:25.824267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e344159c-15b8-49cb-952e-76db3967db18 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:25.925052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e344159c-15b8-49cb-952e-76db3967db18 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:25.948647 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:25.960129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6cb922-1a39-499c-83d3-fb5c8acd7f50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:25.960728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6cb922-1a39-499c-83d3-fb5c8acd7f50 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:25.961005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6cb922-1a39-499c-83d3-fb5c8acd7f50 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:25.961276 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6cb922-1a39-499c-83d3-fb5c8acd7f50 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:25.962081 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6cb922-1a39-499c-83d3-fb5c8acd7f50 None admin] RBAC: Adding query filter params (type=image) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:25.962385 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6cb922-1a39-499c-83d3-fb5c8acd7f50 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:25.963788 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6cb922-1a39-499c-83d3-fb5c8acd7f50 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:25.973499 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 520/1025] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:25 2022] GET /identity/v3/services?type=image => generated 350 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:25.974686 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307d3e7a-d476-43c9-b1e9-8bab0108eae3 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:26.075471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307d3e7a-d476-43c9-b1e9-8bab0108eae3 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:26.109389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff6c2c2-f538-4de3-b22a-aefc210dd742 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:26.109994 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff6c2c2-f538-4de3-b22a-aefc210dd742 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:26.110270 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff6c2c2-f538-4de3-b22a-aefc210dd742 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:26.110534 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff6c2c2-f538-4de3-b22a-aefc210dd742 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:26.111245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff6c2c2-f538-4de3-b22a-aefc210dd742 None admin] RBAC: Adding query filter params (interface=public, service_id=982b81e14d7448589311a2012e01818a, region_id=RegionOne) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:26.111550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff6c2c2-f538-4de3-b22a-aefc210dd742 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:26.112924 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff6c2c2-f538-4de3-b22a-aefc210dd742 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:26.123278 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 506/1026] 10.222.0.39 () {56 vars in 1251 bytes} [Fri Sep 30 09:28:25 2022] GET /identity/v3/endpoints?service_id=982b81e14d7448589311a2012e01818a&interface=public®ion_id=RegionOne => generated 194 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:26.124119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6cb922-1a39-499c-83d3-fb5c8acd7f50 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:26.236668 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6cb922-1a39-499c-83d3-fb5c8acd7f50 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:26.252109 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf0bb9f-a23d-43c0-a9f0-ad45dd1b4dfb None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:26.252388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf0bb9f-a23d-43c0-a9f0-ad45dd1b4dfb None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:26.252649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf0bb9f-a23d-43c0-a9f0-ad45dd1b4dfb None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:26.254210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 507/1027] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:28:26.259162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e67704-d08a-440b-880a-5164e7f7ad9a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:26.259456 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e67704-d08a-440b-880a-5164e7f7ad9a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:26.259726 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e67704-d08a-440b-880a-5164e7f7ad9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:26.273296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55f7357-910f-4132-ab66-10d201a0d841 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:26.273898 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55f7357-910f-4132-ab66-10d201a0d841 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:26.274183 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55f7357-910f-4132-ab66-10d201a0d841 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:26.274480 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55f7357-910f-4132-ab66-10d201a0d841 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:26.275109 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55f7357-910f-4132-ab66-10d201a0d841 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:26.276472 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55f7357-910f-4132-ab66-10d201a0d841 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:26.277906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-03e67704-d08a-440b-880a-5164e7f7ad9a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:26.279107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-03e67704-d08a-440b-880a-5164e7f7ad9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:26.290846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 521/1028] 10.222.0.39 () {56 vars in 1087 bytes} [Fri Sep 30 09:28:26 2022] GET /identity/v3/services => generated 1957 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:28:26.298119 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:26.357314 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 508/1029] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:26 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 30 09:28:26.621883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55f7357-910f-4132-ab66-10d201a0d841 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:26.675082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:26.725828 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55f7357-910f-4132-ab66-10d201a0d841 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:26.762526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d539955-e8d0-4639-a3a1-1a6d960f59b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:26.763551 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d539955-e8d0-4639-a3a1-1a6d960f59b0 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:26.764053 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d539955-e8d0-4639-a3a1-1a6d960f59b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:26.764542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d539955-e8d0-4639-a3a1-1a6d960f59b0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:26.765841 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d539955-e8d0-4639-a3a1-1a6d960f59b0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:26.767540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d539955-e8d0-4639-a3a1-1a6d960f59b0 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:26.769711 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 522/1030] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:26 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:26.775575 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e67704-d08a-440b-880a-5164e7f7ad9a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:26.878961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e67704-d08a-440b-880a-5164e7f7ad9a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:26.916015 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e65201d-fc32-4727-84db-fad7318466fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:26.916650 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e65201d-fc32-4727-84db-fad7318466fb None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:26.917041 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e65201d-fc32-4727-84db-fad7318466fb None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:26.917235 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e65201d-fc32-4727-84db-fad7318466fb None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:26.917900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e65201d-fc32-4727-84db-fad7318466fb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:26.919422 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e65201d-fc32-4727-84db-fad7318466fb None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:27.015851 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15dd9a6-7c75-4f49-991d-06b982f745ef None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:27.016188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15dd9a6-7c75-4f49-991d-06b982f745ef None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:27.016399 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15dd9a6-7c75-4f49-991d-06b982f745ef None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:27.021584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 509/1031] 10.222.0.39 () {60 vars in 1132 bytes} [Fri Sep 30 09:28:26 2022] POST /identity/v3/users => generated 285 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:28:27.021753 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 523/1032] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:28:27.023330 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9034dc-f8dc-4368-9973-aba014fdfc0a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:27.023628 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9034dc-f8dc-4368-9973-aba014fdfc0a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:27.023913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9034dc-f8dc-4368-9973-aba014fdfc0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:27.032043 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:27.044193 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-6f9034dc-f8dc-4368-9973-aba014fdfc0a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:27.045439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6f9034dc-f8dc-4368-9973-aba014fdfc0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:27.121041 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 510/1033] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:27 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:27.391786 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15dd9a6-7c75-4f49-991d-06b982f745ef None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:27.497617 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15dd9a6-7c75-4f49-991d-06b982f745ef None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:27.516524 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:27.533575 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e9bee4-758f-4d6e-a0e5-cc3e798bbeac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:27.534213 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e9bee4-758f-4d6e-a0e5-cc3e798bbeac None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:27.534495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e9bee4-758f-4d6e-a0e5-cc3e798bbeac None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:27.534791 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e9bee4-758f-4d6e-a0e5-cc3e798bbeac None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:27.545130 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e9bee4-758f-4d6e-a0e5-cc3e798bbeac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:28:27.550019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-e8e9bee4-758f-4d6e-a0e5-cc3e798bbeac None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 30 09:28:27.550019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:27.550019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:27.550019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:27.550019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:27.550019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:27.550019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:27.550019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:27.550019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:27.550019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:27.550019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:27.550019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:27.550019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:28:27.550019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:28:27.550019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:28:27.550019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:28:27.551184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:28:27.551184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:28:27.551184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:28:27.551184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:28:27.551184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:27.551184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:27.551184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:27.551184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:27.551184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:27.551184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:27.551184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:27.551184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:27.551184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:27.551184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:27.551184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:27.551184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:27.552187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:27.552187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:27.552187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:27.552187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:27.552187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:28:27.552187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:28:27.552187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:28:27.552187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:28:27.552187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:28:27.552187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:28:27.552187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 30 09:28:27.552187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:27.554689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 524/1034] 10.222.0.39 () {56 vars in 1120 bytes} [Fri Sep 30 09:28:27 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:27.557718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9034dc-f8dc-4368-9973-aba014fdfc0a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:27.657891 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9034dc-f8dc-4368-9973-aba014fdfc0a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:27.692591 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaae31e1-9055-46c7-a574-8822e073c9b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:27.693205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaae31e1-9055-46c7-a574-8822e073c9b5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:27.693490 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaae31e1-9055-46c7-a574-8822e073c9b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:27.693767 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaae31e1-9055-46c7-a574-8822e073c9b5 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:27.694424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaae31e1-9055-46c7-a574-8822e073c9b5 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:27.694759 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaae31e1-9055-46c7-a574-8822e073c9b5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:27.696147 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaae31e1-9055-46c7-a574-8822e073c9b5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:27.697551 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:27.709644 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 511/1035] 10.222.0.39 () {56 vars in 1115 bytes} [Fri Sep 30 09:28:27 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:27.710835 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e9bee4-758f-4d6e-a0e5-cc3e798bbeac None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:27.813352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e9bee4-758f-4d6e-a0e5-cc3e798bbeac None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:27.846845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c72a6bf-9c23-4896-936b-d89c0bc61b07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:27.847481 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c72a6bf-9c23-4896-936b-d89c0bc61b07 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:27.847763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c72a6bf-9c23-4896-936b-d89c0bc61b07 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:27.848034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c72a6bf-9c23-4896-936b-d89c0bc61b07 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:27.858702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c72a6bf-9c23-4896-936b-d89c0bc61b07 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:27.860243 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c72a6bf-9c23-4896-936b-d89c0bc61b07 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:27.875518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-8c72a6bf-9c23-4896-936b-d89c0bc61b07 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:27.875518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:27.875518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:27.875518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:27.875518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:27.875518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:27.875518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:27.875518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:27.875518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:27.875518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:27.875518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:27.875518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:27.875518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:28:27.875518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:28:27.875518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:28:27.875518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:28:27.876529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:27.876529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:27.876529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:27.876529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:27.876529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:27.876529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:27.876529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:27.876529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:27.876529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:27.876529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:27.876529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:27.876529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:27.876529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:27.876529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:27.876529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:27.876529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:27.876529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:28:27.877470 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:28:27.877470 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:27.877470 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:27.877470 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 525/1036] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:27.884043 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaae31e1-9055-46c7-a574-8822e073c9b5 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:27.986582 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaae31e1-9055-46c7-a574-8822e073c9b5 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:28.020053 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e243aa31-5130-4dd6-88bc-b4531ae0d91c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:28.020665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e243aa31-5130-4dd6-88bc-b4531ae0d91c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:28.020948 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e243aa31-5130-4dd6-88bc-b4531ae0d91c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:28.021223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e243aa31-5130-4dd6-88bc-b4531ae0d91c None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:28.021859 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e243aa31-5130-4dd6-88bc-b4531ae0d91c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:28.022170 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e243aa31-5130-4dd6-88bc-b4531ae0d91c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:28.023611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e243aa31-5130-4dd6-88bc-b4531ae0d91c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:28.040502 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 512/1037] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:28:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:28.046386 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c72a6bf-9c23-4896-936b-d89c0bc61b07 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:28.077333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:28.148888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c72a6bf-9c23-4896-936b-d89c0bc61b07 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:28.188771 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a9c704-05e6-46a0-bd23-9e382edd2f8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:28.189415 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a9c704-05e6-46a0-bd23-9e382edd2f8d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:28.189693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a9c704-05e6-46a0-bd23-9e382edd2f8d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:28.189965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a9c704-05e6-46a0-bd23-9e382edd2f8d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:28.197558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a9c704-05e6-46a0-bd23-9e382edd2f8d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:28.199037 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a9c704-05e6-46a0-bd23-9e382edd2f8d None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:28.210890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-82a9c704-05e6-46a0-bd23-9e382edd2f8d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 30 09:28:28.210890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:28.210890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:28.210890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:28.210890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:28.210890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:28.210890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:28.210890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:28.210890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:28.210890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:28.210890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:28.210890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:28.210890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:28:28.210890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:28:28.210890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:28:28.210890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:28:28.212377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:28.212377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:28.212377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:28:28.212377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:28:28.212377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:28:28.212377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:28:28.212377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:28.212377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:28.212377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:28.212377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:28.212377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:28.212377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:28.212377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:28.212377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:28.212377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:28.212377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:28.212377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:28.212377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:28.218492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:28.218492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:28.218492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:28:28.218492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:28:28.218492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:28:28.218492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:28:28.218492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:28:28.218492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:28:28.218492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 30 09:28:28.218492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:28.218492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 526/1038] 10.222.0.39 () {56 vars in 1093 bytes} [Fri Sep 30 09:28:28 2022] GET /identity/v3/users/nova => generated 82 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:28.220790 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e243aa31-5130-4dd6-88bc-b4531ae0d91c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:28.279601 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:28.324680 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e243aa31-5130-4dd6-88bc-b4531ae0d91c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:28.363181 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710873be-7580-43a3-89aa-eee742a5f645 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:28.363703 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710873be-7580-43a3-89aa-eee742a5f645 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:28.363984 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710873be-7580-43a3-89aa-eee742a5f645 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:28.364259 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710873be-7580-43a3-89aa-eee742a5f645 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:28.364987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710873be-7580-43a3-89aa-eee742a5f645 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:28.365294 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710873be-7580-43a3-89aa-eee742a5f645 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:28.366751 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710873be-7580-43a3-89aa-eee742a5f645 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:28.386321 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 513/1039] 10.222.0.39 () {56 vars in 1133 bytes} [Fri Sep 30 09:28:28 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:28.392159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a9c704-05e6-46a0-bd23-9e382edd2f8d None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:28.453785 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73312315-e582-4458-b050-d553f4be86ce None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:28.454100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73312315-e582-4458-b050-d553f4be86ce None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:28.454342 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73312315-e582-4458-b050-d553f4be86ce None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:28.455836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 514/1040] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:28:28.460452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e446659-7e64-4538-aa73-4a1b4ea9fb0b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:28.460742 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e446659-7e64-4538-aa73-4a1b4ea9fb0b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:28.461028 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e446659-7e64-4538-aa73-4a1b4ea9fb0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:28.479301 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-6e446659-7e64-4538-aa73-4a1b4ea9fb0b None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:28.480487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6e446659-7e64-4538-aa73-4a1b4ea9fb0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:28.488320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:28.500082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a9c704-05e6-46a0-bd23-9e382edd2f8d None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:28.537679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3aeafae-54dc-4659-a35a-79cb712b244a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:28.538283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3aeafae-54dc-4659-a35a-79cb712b244a None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:28.538621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3aeafae-54dc-4659-a35a-79cb712b244a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:28.538863 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3aeafae-54dc-4659-a35a-79cb712b244a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:28.552830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 515/1041] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:28 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 30 09:28:28.609353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3aeafae-54dc-4659-a35a-79cb712b244a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:28.610853 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3aeafae-54dc-4659-a35a-79cb712b244a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:28.700238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:28.715418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 527/1042] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:28:28 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 30 09:28:28.717182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e446659-7e64-4538-aa73-4a1b4ea9fb0b None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:28.765325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:28.819337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e446659-7e64-4538-aa73-4a1b4ea9fb0b None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:28.823718 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3aeafae-54dc-4659-a35a-79cb712b244a None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:28.856499 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f03559-9f02-4c62-826c-fe7fb8a419b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:28.857128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f03559-9f02-4c62-826c-fe7fb8a419b2 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:28.857404 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f03559-9f02-4c62-826c-fe7fb8a419b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:28.857673 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f03559-9f02-4c62-826c-fe7fb8a419b2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:28.868341 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f03559-9f02-4c62-826c-fe7fb8a419b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:28.869789 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f03559-9f02-4c62-826c-fe7fb8a419b2 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:28.885271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-12f03559-9f02-4c62-826c-fe7fb8a419b2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:28.885271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:28.885271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:28.885271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:28.885271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:28.885271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:28.885271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:28.885271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:28.885271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:28.885271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:28.885271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:28.885271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:28.885271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:28:28.885271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:28:28.885271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:28:28.885271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:28:28.886961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:28.886961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:28.886961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:28.886961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:28.886961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:28.886961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:28.886961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:28.886961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:28.886961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:28.886961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:28.886961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:28.886961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:28.886961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:28.886961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:28.886961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:28.886961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:28.886961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:28:28.887912 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:28:28.887912 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:28.887912 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:28.887912 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 516/1043] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:28.892136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f03559-9f02-4c62-826c-fe7fb8a419b2 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:28.934075 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3aeafae-54dc-4659-a35a-79cb712b244a None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:28.972110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943032bd-90f6-4788-a426-48ef086d00a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:28.972759 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943032bd-90f6-4788-a426-48ef086d00a5 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:28.973338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943032bd-90f6-4788-a426-48ef086d00a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:28.973706 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943032bd-90f6-4788-a426-48ef086d00a5 None admin] PATH_INFO: `/v3/services/image` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:28.974202 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943032bd-90f6-4788-a426-48ef086d00a5 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:28.975614 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943032bd-90f6-4788-a426-48ef086d00a5 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:28.984633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-943032bd-90f6-4788-a426-48ef086d00a5 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 30 09:28:28.984633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:28.984633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:28:28.984633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:28:28.984633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:28:28.984633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:28:28.984633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:28:28.984633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:28.984633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 30 09:28:28.984633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:28.984633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:28:28.984633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:28.984633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:28.984633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:28.984633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:28.984633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:28.984633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:28.985730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:28.985730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:28.985730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:28.985730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:28.985730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:28.985730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:28.985730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:28:28.985730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:28:28.985730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:28:28.985730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:28:28.985730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:28.985730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:28.985730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:28.985730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:28.985730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:28.985730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:28.986670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:28.986670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:28.986670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:28.986670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:28.986670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:28.986670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:28.986670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:28.986670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:28.986670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:28.986670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:28.986670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:28:28.986670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:28:28.986670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 30 09:28:28.986670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:28.986670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 528/1044] 10.222.0.39 () {56 vars in 1105 bytes} [Fri Sep 30 09:28:28 2022] GET /identity/v3/services/image => generated 86 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:28.991617 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943032bd-90f6-4788-a426-48ef086d00a5 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:28.992709 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:29.000075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f03559-9f02-4c62-826c-fe7fb8a419b2 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:29.039466 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878f647e-a12f-40d2-be8b-03d36ef89976 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:29.040365 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878f647e-a12f-40d2-be8b-03d36ef89976 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:29.040649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878f647e-a12f-40d2-be8b-03d36ef89976 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:29.041109 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878f647e-a12f-40d2-be8b-03d36ef89976 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:29.042029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878f647e-a12f-40d2-be8b-03d36ef89976 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:29.042346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878f647e-a12f-40d2-be8b-03d36ef89976 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:29.043778 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878f647e-a12f-40d2-be8b-03d36ef89976 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:29.047155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:29.057064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 517/1045] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:28:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:29.062410 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878f647e-a12f-40d2-be8b-03d36ef89976 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:29.095389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943032bd-90f6-4788-a426-48ef086d00a5 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:29.131629 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3a86a1-da25-406f-b719-fe0e7e593bf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:29.132272 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3a86a1-da25-406f-b719-fe0e7e593bf2 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:29.132557 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3a86a1-da25-406f-b719-fe0e7e593bf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:29.132834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3a86a1-da25-406f-b719-fe0e7e593bf2 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:29.133514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3a86a1-da25-406f-b719-fe0e7e593bf2 None admin] RBAC: Adding query filter params (name=image) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:29.133824 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3a86a1-da25-406f-b719-fe0e7e593bf2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:29.135304 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3a86a1-da25-406f-b719-fe0e7e593bf2 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:29.145676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 529/1046] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:28 2022] GET /identity/v3/services?name=image => generated 122 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:29.146915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3a86a1-da25-406f-b719-fe0e7e593bf2 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:29.177252 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878f647e-a12f-40d2-be8b-03d36ef89976 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:29.216007 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c85909-2f25-44ad-8e70-f9399e9669a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:29.216727 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c85909-2f25-44ad-8e70-f9399e9669a7 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:29.217015 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c85909-2f25-44ad-8e70-f9399e9669a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:29.217289 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c85909-2f25-44ad-8e70-f9399e9669a7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:29.224235 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c85909-2f25-44ad-8e70-f9399e9669a7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:29.225610 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c85909-2f25-44ad-8e70-f9399e9669a7 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:29.238295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-33c85909-2f25-44ad-8e70-f9399e9669a7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:28:29.238295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:29.238295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:29.238295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:29.238295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:29.238295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:29.238295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:29.238295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:29.238295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:29.238295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:29.238295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:29.238295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:29.238295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:28:29.238295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:28:29.238295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:28:29.239452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:28:29.239452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:29.239452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:29.239452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:29.239452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:29.239452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:29.239452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:29.239452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:29.239452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:29.239452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:29.239452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:29.239452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:29.239452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:29.239452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:29.239452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:29.239452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:29.239452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:29.240921 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:28:29.240921 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:28:29.240921 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:28:29.240921 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:28:29.240921 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:28:29.240921 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:28:29.240921 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:28:29.240921 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:29.240921 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 518/1047] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:28:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:29.245960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bb9d6c-5cf9-430e-a10f-9e0a0924bad7 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:29.246226 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bb9d6c-5cf9-430e-a10f-9e0a0924bad7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:29.246502 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bb9d6c-5cf9-430e-a10f-9e0a0924bad7 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:29.248320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 519/1048] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:28:29.249017 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bb9d6c-5cf9-430e-a10f-9e0a0924bad7 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:29.259704 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3a86a1-da25-406f-b719-fe0e7e593bf2 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:29.267443 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:29.276325 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:29.293295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e9dd5c-9669-404b-8c0b-94fedcc0ab8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:29.293894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e9dd5c-9669-404b-8c0b-94fedcc0ab8d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:29.294175 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e9dd5c-9669-404b-8c0b-94fedcc0ab8d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:29.294450 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e9dd5c-9669-404b-8c0b-94fedcc0ab8d None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:29.295097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e9dd5c-9669-404b-8c0b-94fedcc0ab8d None admin] RBAC: Adding query filter params (type=image) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:29.295401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e9dd5c-9669-404b-8c0b-94fedcc0ab8d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:29.296822 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e9dd5c-9669-404b-8c0b-94fedcc0ab8d None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:29.306831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 530/1049] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:29 2022] GET /identity/v3/services?type=image => generated 350 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:29.306831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb74366-1c13-468d-8373-fbe2a899489c None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:29.306831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb74366-1c13-468d-8373-fbe2a899489c None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:29.306831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb74366-1c13-468d-8373-fbe2a899489c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:29.321807 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-dcb74366-1c13-468d-8373-fbe2a899489c None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:29.323035 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-dcb74366-1c13-468d-8373-fbe2a899489c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:29.351100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bb9d6c-5cf9-430e-a10f-9e0a0924bad7 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:29.386204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3133f4-7938-4494-805c-2f5ce3ae9f2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:29.386838 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3133f4-7938-4494-805c-2f5ce3ae9f2b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:29.387142 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3133f4-7938-4494-805c-2f5ce3ae9f2b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:29.387415 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3133f4-7938-4494-805c-2f5ce3ae9f2b None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:29.388089 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3133f4-7938-4494-805c-2f5ce3ae9f2b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:29.388390 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3133f4-7938-4494-805c-2f5ce3ae9f2b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:29.388646 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 531/1050] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:29 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:29.389448 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb74366-1c13-468d-8373-fbe2a899489c None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:29.389880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3133f4-7938-4494-805c-2f5ce3ae9f2b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:29.406478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 520/1051] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:28:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:29.407317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3133f4-7938-4494-805c-2f5ce3ae9f2b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:29.489622 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:29.495862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb74366-1c13-468d-8373-fbe2a899489c None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:29.510132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:29.513261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3133f4-7938-4494-805c-2f5ce3ae9f2b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:29.530742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709ab973-0492-4060-bf86-a8100af253ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:29.531339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709ab973-0492-4060-bf86-a8100af253ce None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:29.531617 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709ab973-0492-4060-bf86-a8100af253ce None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:29.531886 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709ab973-0492-4060-bf86-a8100af253ce None admin] PATH_INFO: `/v3/endpoints` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:29.532539 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709ab973-0492-4060-bf86-a8100af253ce None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:29.533901 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709ab973-0492-4060-bf86-a8100af253ce None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:29.550547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd9972e-e6e7-46cf-ac13-4703a987b944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:29.551176 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd9972e-e6e7-46cf-ac13-4703a987b944 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:29.551457 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd9972e-e6e7-46cf-ac13-4703a987b944 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:29.551729 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd9972e-e6e7-46cf-ac13-4703a987b944 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:29.552438 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd9972e-e6e7-46cf-ac13-4703a987b944 None admin] RBAC: Adding query filter params (role.id=fa71d6ff1b04452c98c5830f178feb1b, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=7ff68fb87700423ab53c90520a6bd3b9) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:29.552802 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd9972e-e6e7-46cf-ac13-4703a987b944 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:29.554324 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd9972e-e6e7-46cf-ac13-4703a987b944 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:29.559717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 521/1052] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:28:29 2022] GET /identity/v3/role_assignments?user.id=7ff68fb87700423ab53c90520a6bd3b9&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=fa71d6ff1b04452c98c5830f178feb1b => generated 259 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:29.646642 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 532/1053] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:28:29 2022] POST /identity/v3/endpoints => generated 327 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:28:29.656791 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd9972e-e6e7-46cf-ac13-4703a987b944 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:29.779795 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd9972e-e6e7-46cf-ac13-4703a987b944 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:29.822985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628ac054-d60c-404e-ac15-75f93e2e29fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:29.823618 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628ac054-d60c-404e-ac15-75f93e2e29fc None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:29.823892 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628ac054-d60c-404e-ac15-75f93e2e29fc None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:29.824162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628ac054-d60c-404e-ac15-75f93e2e29fc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:29.826906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:29.835823 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628ac054-d60c-404e-ac15-75f93e2e29fc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:28:29.841237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-628ac054-d60c-404e-ac15-75f93e2e29fc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:28:29.841237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:29.841237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:29.841237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:29.841237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:29.841237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:29.841237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:29.841237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:29.841237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:29.841237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:29.841237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:29.841237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:29.841237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:28:29.841237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:28:29.841237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:28:29.841237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:28:29.842204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:28:29.842204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:28:29.842204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:28:29.842204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:28:29.842204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:29.842204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:29.842204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:29.842204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:29.842204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:29.842204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:29.842204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:29.842204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:29.842204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:29.842204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:29.842204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:29.842204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:29.843097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:29.843097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:29.843097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:29.843097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:29.843097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:28:29.843097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:28:29.843097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:28:29.843097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:28:29.843097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:28:29.843097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:28:29.843097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:28:29.843097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:29.843097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 522/1054] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:28:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:29.848600 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709ab973-0492-4060-bf86-a8100af253ce None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:29.954285 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709ab973-0492-4060-bf86-a8100af253ce None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:29.988359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5928b1fc-5b2d-4526-b4b2-cf6b480f69ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:29.988971 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5928b1fc-5b2d-4526-b4b2-cf6b480f69ac None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:29.989253 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5928b1fc-5b2d-4526-b4b2-cf6b480f69ac None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:29.989526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5928b1fc-5b2d-4526-b4b2-cf6b480f69ac None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:29.990182 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5928b1fc-5b2d-4526-b4b2-cf6b480f69ac None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:29.990487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5928b1fc-5b2d-4526-b4b2-cf6b480f69ac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:29.991921 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5928b1fc-5b2d-4526-b4b2-cf6b480f69ac None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:30.005907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 533/1055] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:28:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:30.007112 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628ac054-d60c-404e-ac15-75f93e2e29fc None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:30.111471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628ac054-d60c-404e-ac15-75f93e2e29fc None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:30.145139 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3d19ed-4f06-4c4f-84fc-18abcb9e24a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:30.145768 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3d19ed-4f06-4c4f-84fc-18abcb9e24a3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:30.146051 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3d19ed-4f06-4c4f-84fc-18abcb9e24a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:30.146317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3d19ed-4f06-4c4f-84fc-18abcb9e24a3 None admin] PATH_INFO: `/v3/users/f0a9ae0ae9a04d16b7d7baf8d840838e` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:30.160272 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3d19ed-4f06-4c4f-84fc-18abcb9e24a3 None admin] RBAC: Authorizing `identity:get_user(user_id=f0a9ae0ae9a04d16b7d7baf8d840838e)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:30.161705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3d19ed-4f06-4c4f-84fc-18abcb9e24a3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:30.163520 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 523/1056] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:28:30 2022] GET /identity/v3/users/f0a9ae0ae9a04d16b7d7baf8d840838e => generated 285 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:30.169226 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5928b1fc-5b2d-4526-b4b2-cf6b480f69ac None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:30.207068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:30.279173 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5928b1fc-5b2d-4526-b4b2-cf6b480f69ac None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:30.313905 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d6bc21-56d6-41c4-8a2b-5fa8386ab801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:30.314513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d6bc21-56d6-41c4-8a2b-5fa8386ab801 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:30.314821 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d6bc21-56d6-41c4-8a2b-5fa8386ab801 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:30.315098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d6bc21-56d6-41c4-8a2b-5fa8386ab801 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:30.384189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d6bc21-56d6-41c4-8a2b-5fa8386ab801 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:30.385579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d6bc21-56d6-41c4-8a2b-5fa8386ab801 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:30.414397 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:30.493164 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 534/1057] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:28:30 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 30 09:28:30.494476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3d19ed-4f06-4c4f-84fc-18abcb9e24a3 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:30.553303 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:30.598003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3d19ed-4f06-4c4f-84fc-18abcb9e24a3 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:30.631537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4f96a6-529e-4c32-8561-44b1b5cff48f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:30.632162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4f96a6-529e-4c32-8561-44b1b5cff48f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:30.632443 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4f96a6-529e-4c32-8561-44b1b5cff48f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:30.632716 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4f96a6-529e-4c32-8561-44b1b5cff48f None admin] PATH_INFO: `/v3/projects/d44d638c1b00415c8d972b575a7bd2b7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:30.633847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4f96a6-529e-4c32-8561-44b1b5cff48f None admin] RBAC: Authorizing `identity:get_project(project_id=d44d638c1b00415c8d972b575a7bd2b7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:30.635350 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4f96a6-529e-4c32-8561-44b1b5cff48f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:30.637134 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 524/1058] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:28:30 2022] GET /identity/v3/projects/d44d638c1b00415c8d972b575a7bd2b7 => generated 311 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:30.643005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d6bc21-56d6-41c4-8a2b-5fa8386ab801 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:30.744238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d6bc21-56d6-41c4-8a2b-5fa8386ab801 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:30.772600 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:30.779137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3dee5b-1219-47fb-8018-7d57531391b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:30.779843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3dee5b-1219-47fb-8018-7d57531391b8 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:30.780176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3dee5b-1219-47fb-8018-7d57531391b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:30.780511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3dee5b-1219-47fb-8018-7d57531391b8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:30.781296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3dee5b-1219-47fb-8018-7d57531391b8 None admin] RBAC: Adding query filter params (role.id=e874c311084c4c9fb7691bce461d319b, scope.project.id=d44d638c1b00415c8d972b575a7bd2b7, user.id=f0a9ae0ae9a04d16b7d7baf8d840838e) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:30.792233 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3dee5b-1219-47fb-8018-7d57531391b8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:30.792233 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3dee5b-1219-47fb-8018-7d57531391b8 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:30.792233 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 535/1059] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:28:30 2022] GET /identity/v3/role_assignments?user.id=f0a9ae0ae9a04d16b7d7baf8d840838e&scope.project.id=d44d638c1b00415c8d972b575a7bd2b7&role.id=e874c311084c4c9fb7691bce461d319b => generated 259 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:31.739529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b237066-eee6-49da-a071-0ba5dc2fafd3 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:31.740093 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b237066-eee6-49da-a071-0ba5dc2fafd3 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:31.740093 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b237066-eee6-49da-a071-0ba5dc2fafd3 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:31.741447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 525/1060] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:31.747382 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b0d162-bfb3-466b-809d-5ea066b09ba6 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:31.747763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b0d162-bfb3-466b-809d-5ea066b09ba6 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:31.747923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b0d162-bfb3-466b-809d-5ea066b09ba6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:31.766836 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-52b0d162-bfb3-466b-809d-5ea066b09ba6 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:31.768019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-52b0d162-bfb3-466b-809d-5ea066b09ba6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:31.829754 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12a9e0d-fa67-4e8b-b8e1-71d13fa861b4 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:31.830037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12a9e0d-fa67-4e8b-b8e1-71d13fa861b4 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:31.830302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12a9e0d-fa67-4e8b-b8e1-71d13fa861b4 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:31.831808 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 526/1061] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:28:31.835982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 536/1062] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:31.836750 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ddcd15-66aa-4967-923b-c276e9496b98 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:31.837135 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ddcd15-66aa-4967-923b-c276e9496b98 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:31.837422 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ddcd15-66aa-4967-923b-c276e9496b98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:31.855401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-26ddcd15-66aa-4967-923b-c276e9496b98 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:31.856584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-26ddcd15-66aa-4967-923b-c276e9496b98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:31.925075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 527/1063] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 30 09:28:32.108589 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b0d162-bfb3-466b-809d-5ea066b09ba6 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:32.188143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ddcd15-66aa-4967-923b-c276e9496b98 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:32.213066 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b0d162-bfb3-466b-809d-5ea066b09ba6 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:32.238150 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:32.245811 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:32.248956 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa30771-ab9b-4eb6-8ebc-fd22f8b3b52a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:32.249590 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa30771-ab9b-4eb6-8ebc-fd22f8b3b52a None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:32.259991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa30771-ab9b-4eb6-8ebc-fd22f8b3b52a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:32.259991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa30771-ab9b-4eb6-8ebc-fd22f8b3b52a None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:32.261051 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa30771-ab9b-4eb6-8ebc-fd22f8b3b52a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:28:32.265854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-cfa30771-ab9b-4eb6-8ebc-fd22f8b3b52a None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 30 09:28:32.265854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:32.265854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:32.265854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:32.265854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:32.265854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:32.265854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:32.265854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:32.265854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:32.265854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:32.265854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:32.265854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:32.265854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:28:32.265854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:28:32.265854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:28:32.265854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:28:32.267052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:28:32.267052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:28:32.267052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:28:32.267052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:28:32.267052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:32.267052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:32.267052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:32.267052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:32.267052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:32.267052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:32.267052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:32.267052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:32.267052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:32.267052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:32.267052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:32.267052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:32.267946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:32.267946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:32.267946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:32.267946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:32.267946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:28:32.267946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:28:32.267946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:28:32.267946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:28:32.267946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:28:32.267946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:28:32.267946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 30 09:28:32.267946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:32.267946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 537/1064] 10.222.0.39 () {56 vars in 1120 bytes} [Fri Sep 30 09:28:32 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:32.273545 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa30771-ab9b-4eb6-8ebc-fd22f8b3b52a None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:32.299038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ddcd15-66aa-4967-923b-c276e9496b98 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:32.335774 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8868de4-c516-43dc-8e4e-5492cb4ddc34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:32.337028 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8868de4-c516-43dc-8e4e-5492cb4ddc34 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:32.337312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8868de4-c516-43dc-8e4e-5492cb4ddc34 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:32.337591 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8868de4-c516-43dc-8e4e-5492cb4ddc34 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:32.377484 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa30771-ab9b-4eb6-8ebc-fd22f8b3b52a None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:32.407763 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8868de4-c516-43dc-8e4e-5492cb4ddc34 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:32.409285 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8868de4-c516-43dc-8e4e-5492cb4ddc34 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:32.412246 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4724744-744e-4671-a7fe-7d1012e6ffcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:32.412850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4724744-744e-4671-a7fe-7d1012e6ffcc None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:32.413133 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4724744-744e-4671-a7fe-7d1012e6ffcc None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:32.413405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4724744-744e-4671-a7fe-7d1012e6ffcc None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:32.414066 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4724744-744e-4671-a7fe-7d1012e6ffcc None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:32.414396 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4724744-744e-4671-a7fe-7d1012e6ffcc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:32.415841 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4724744-744e-4671-a7fe-7d1012e6ffcc None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:32.428690 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 538/1065] 10.222.0.39 () {56 vars in 1115 bytes} [Fri Sep 30 09:28:32 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:32.429741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4724744-744e-4671-a7fe-7d1012e6ffcc None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:32.469550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:32.485731 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:32.523122 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 528/1066] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:28:32 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 30 09:28:32.528923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8868de4-c516-43dc-8e4e-5492cb4ddc34 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:32.544399 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4724744-744e-4671-a7fe-7d1012e6ffcc None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:32.579373 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afbf9c5-c278-4040-a5a6-b7d980bb0a96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:32.580107 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afbf9c5-c278-4040-a5a6-b7d980bb0a96 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:32.580392 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afbf9c5-c278-4040-a5a6-b7d980bb0a96 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:32.580670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afbf9c5-c278-4040-a5a6-b7d980bb0a96 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:32.591534 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9afbf9c5-c278-4040-a5a6-b7d980bb0a96 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:32.592983 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9afbf9c5-c278-4040-a5a6-b7d980bb0a96 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:32.607360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-9afbf9c5-c278-4040-a5a6-b7d980bb0a96 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:32.607360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:32.607360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:32.607360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:32.607360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:32.607360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:32.607360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:32.607360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:32.607360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:32.607360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:32.607360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:32.607360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:32.607360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:28:32.607360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:28:32.607360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:28:32.607360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:28:32.608348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:32.608348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:32.608348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:32.608348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:32.608348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:32.608348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:32.608348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:32.608348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:32.608348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:32.608348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:32.608348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:32.608348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:32.608348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:32.608348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:32.608348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:32.608348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:32.608348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:28:32.609283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:28:32.609283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:32.609283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:32.609283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 539/1067] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:32.614162 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afbf9c5-c278-4040-a5a6-b7d980bb0a96 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:32.635461 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8868de4-c516-43dc-8e4e-5492cb4ddc34 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:32.673288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3960256a-36c8-4290-b103-044e2c89c79d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:32.674235 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3960256a-36c8-4290-b103-044e2c89c79d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:32.674522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3960256a-36c8-4290-b103-044e2c89c79d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:32.674983 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3960256a-36c8-4290-b103-044e2c89c79d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:32.685657 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3960256a-36c8-4290-b103-044e2c89c79d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:32.687143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3960256a-36c8-4290-b103-044e2c89c79d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:32.701974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-3960256a-36c8-4290-b103-044e2c89c79d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:32.701974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:32.701974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:32.701974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:32.701974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:32.701974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:32.701974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:32.701974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:32.701974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:32.701974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:32.701974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:32.701974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:32.701974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:28:32.701974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:28:32.701974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:28:32.701974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:28:32.703099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:32.703099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:32.703099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:32.703099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:32.703099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:32.703099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:32.703099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:32.703099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:32.703099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:32.703099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:32.703099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:32.703099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:32.703099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:32.703099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:32.703099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:32.703099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:32.703099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:28:32.704175 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:28:32.704175 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:32.704175 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:32.704175 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 529/1068] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:32.708894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3960256a-36c8-4290-b103-044e2c89c79d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:32.713413 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:32.717403 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:32.720994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afbf9c5-c278-4040-a5a6-b7d980bb0a96 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:32.759786 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c0c5b2-2241-4adc-940e-9b33c656b2cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:32.760442 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c0c5b2-2241-4adc-940e-9b33c656b2cb None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:32.760687 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c0c5b2-2241-4adc-940e-9b33c656b2cb None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:32.761081 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c0c5b2-2241-4adc-940e-9b33c656b2cb None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:32.762014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c0c5b2-2241-4adc-940e-9b33c656b2cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:32.762326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c0c5b2-2241-4adc-940e-9b33c656b2cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:32.763737 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c0c5b2-2241-4adc-940e-9b33c656b2cb None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:32.785041 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 540/1069] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:28:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:32.786367 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c0c5b2-2241-4adc-940e-9b33c656b2cb None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:32.817602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3960256a-36c8-4290-b103-044e2c89c79d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:32.855316 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68230b9-ecbd-4313-a220-6e0d9cf44e4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:32.855920 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68230b9-ecbd-4313-a220-6e0d9cf44e4b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:32.856230 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68230b9-ecbd-4313-a220-6e0d9cf44e4b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:32.856468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68230b9-ecbd-4313-a220-6e0d9cf44e4b None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:32.857100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68230b9-ecbd-4313-a220-6e0d9cf44e4b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:32.857402 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68230b9-ecbd-4313-a220-6e0d9cf44e4b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:32.858813 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68230b9-ecbd-4313-a220-6e0d9cf44e4b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:32.871675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 530/1070] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:28:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:32.899745 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c0c5b2-2241-4adc-940e-9b33c656b2cb None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:32.908686 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9fb117-a852-48d6-ba5e-563aa03ed9c4 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:32.909008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9fb117-a852-48d6-ba5e-563aa03ed9c4 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:32.909242 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9fb117-a852-48d6-ba5e-563aa03ed9c4 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:32.910874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 531/1071] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:28:32.916757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddbbb7e-7d98-4fe2-aa34-74962c44d081 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:32.917045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddbbb7e-7d98-4fe2-aa34-74962c44d081 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:32.917317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddbbb7e-7d98-4fe2-aa34-74962c44d081 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:32.937305 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e73636-f192-4e0e-bdb5-61db2dedef3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:32.937684 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-6ddbbb7e-7d98-4fe2-aa34-74962c44d081 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:32.937955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e73636-f192-4e0e-bdb5-61db2dedef3f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:32.938235 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e73636-f192-4e0e-bdb5-61db2dedef3f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:32.938506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e73636-f192-4e0e-bdb5-61db2dedef3f None admin] PATH_INFO: `/v3/users/nova` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:32.938876 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6ddbbb7e-7d98-4fe2-aa34-74962c44d081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:32.947297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e73636-f192-4e0e-bdb5-61db2dedef3f None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:32.948788 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e73636-f192-4e0e-bdb5-61db2dedef3f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:32.962339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-79e73636-f192-4e0e-bdb5-61db2dedef3f None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 30 09:28:32.962339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:32.962339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:32.962339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:32.962339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:32.962339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:32.962339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:32.962339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:32.962339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:32.962339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:32.962339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:32.962339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:32.962339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:28:32.962339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:28:32.962339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:28:32.962339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:28:32.963621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:32.963621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:32.963621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:28:32.963621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:28:32.963621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:28:32.963621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:28:32.963621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:32.963621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:32.963621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:32.963621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:32.963621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:32.963621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:32.963621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:32.963621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:32.963621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:32.963621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:32.963621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:32.963621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:32.964843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:32.964843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:32.964843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:28:32.964843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:28:32.964843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:28:32.964843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:28:32.964843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:28:32.964843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:28:32.964843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 30 09:28:32.964843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:32.966875 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 541/1072] 10.222.0.39 () {56 vars in 1093 bytes} [Fri Sep 30 09:28:32 2022] GET /identity/v3/users/nova => generated 82 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:32.972646 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e73636-f192-4e0e-bdb5-61db2dedef3f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:32.996302 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:33.015112 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:33.024921 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 532/1073] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 30 09:28:33.090743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e73636-f192-4e0e-bdb5-61db2dedef3f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:33.126272 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fdf927-0063-42b4-8ae6-64e24cc62dcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:33.126929 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fdf927-0063-42b4-8ae6-64e24cc62dcc None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:33.127214 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fdf927-0063-42b4-8ae6-64e24cc62dcc None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:33.127490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fdf927-0063-42b4-8ae6-64e24cc62dcc None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:33.128211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fdf927-0063-42b4-8ae6-64e24cc62dcc None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:33.128520 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fdf927-0063-42b4-8ae6-64e24cc62dcc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:33.130221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fdf927-0063-42b4-8ae6-64e24cc62dcc None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:33.148173 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 542/1074] 10.222.0.39 () {56 vars in 1133 bytes} [Fri Sep 30 09:28:32 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:33.154412 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddbbb7e-7d98-4fe2-aa34-74962c44d081 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:33.269687 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddbbb7e-7d98-4fe2-aa34-74962c44d081 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:33.294540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fdf927-0063-42b4-8ae6-64e24cc62dcc None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:33.302806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b17c681-e497-40a2-959e-677cd2192537 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:33.303439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b17c681-e497-40a2-959e-677cd2192537 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:33.303715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b17c681-e497-40a2-959e-677cd2192537 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:33.303986 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b17c681-e497-40a2-959e-677cd2192537 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:33.313793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b17c681-e497-40a2-959e-677cd2192537 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:33.315277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b17c681-e497-40a2-959e-677cd2192537 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:33.329468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-3b17c681-e497-40a2-959e-677cd2192537 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:33.329468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:33.329468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:33.329468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:33.329468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:33.329468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:33.329468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:33.329468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:33.329468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:33.329468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:33.329468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:33.329468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:33.329468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:28:33.329468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:28:33.329468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:28:33.329468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:28:33.330639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:33.330639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:33.330639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:33.330639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:33.330639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:33.330639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:33.330639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:33.330639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:33.330639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:33.330639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:33.330639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:33.330639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:33.330639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:33.330639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:33.330639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:33.330639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:33.330639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:28:33.331572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:28:33.331572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:33.331572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:33.331572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 533/1075] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:33.336721 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b17c681-e497-40a2-959e-677cd2192537 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:33.375795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:33.381983 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:33.398331 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fdf927-0063-42b4-8ae6-64e24cc62dcc None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:33.432805 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b17c681-e497-40a2-959e-677cd2192537 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:33.433125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5a2046-4da3-4a12-a793-c567bd24ace6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:33.433755 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5a2046-4da3-4a12-a793-c567bd24ace6 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:33.434038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5a2046-4da3-4a12-a793-c567bd24ace6 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:33.434343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5a2046-4da3-4a12-a793-c567bd24ace6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:33.444418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5a2046-4da3-4a12-a793-c567bd24ace6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:28:33.449231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-bf5a2046-4da3-4a12-a793-c567bd24ace6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:28:33.449231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:33.449231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:33.449231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:33.449231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:33.449231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:33.449231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:33.449231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:33.449231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:33.449231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:33.449231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:33.449231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:33.449231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:28:33.449231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:28:33.449231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:28:33.449231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:28:33.450270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:28:33.450270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:28:33.450270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:28:33.450270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:28:33.450270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:33.450270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:33.450270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:33.450270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:33.450270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:33.450270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:33.450270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:33.450270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:33.450270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:33.450270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:33.450270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:33.450270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:33.451237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:33.451237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:33.451237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:33.451237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:33.451237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:28:33.451237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:28:33.451237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:28:33.451237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:28:33.451237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:28:33.451237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:28:33.451237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:28:33.451237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:33.451237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 543/1076] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:28:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:33.456249 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5a2046-4da3-4a12-a793-c567bd24ace6 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:33.466664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a859d8c-0913-4438-af0e-73cdf6d070fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:33.467265 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a859d8c-0913-4438-af0e-73cdf6d070fa None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:33.467541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a859d8c-0913-4438-af0e-73cdf6d070fa None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:33.467810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a859d8c-0913-4438-af0e-73cdf6d070fa None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:33.468433 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a859d8c-0913-4438-af0e-73cdf6d070fa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:33.468735 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a859d8c-0913-4438-af0e-73cdf6d070fa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:33.470074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a859d8c-0913-4438-af0e-73cdf6d070fa None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:33.482265 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 534/1077] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:28:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:33.487788 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a859d8c-0913-4438-af0e-73cdf6d070fa None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:33.562209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5a2046-4da3-4a12-a793-c567bd24ace6 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:33.583207 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a859d8c-0913-4438-af0e-73cdf6d070fa None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:33.591710 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:33.596153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b396a293-ae80-48a4-85ae-e65786c2e69d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:33.596752 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b396a293-ae80-48a4-85ae-e65786c2e69d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:33.597031 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b396a293-ae80-48a4-85ae-e65786c2e69d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:33.597301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b396a293-ae80-48a4-85ae-e65786c2e69d None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:33.601592 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b396a293-ae80-48a4-85ae-e65786c2e69d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:33.601592 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b396a293-ae80-48a4-85ae-e65786c2e69d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:33.601592 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b396a293-ae80-48a4-85ae-e65786c2e69d None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:33.602406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:33.612759 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 544/1078] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:28:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:33.613703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b396a293-ae80-48a4-85ae-e65786c2e69d None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:33.619364 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca75f98-79ba-43f9-96be-7de3f10edb61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:33.619990 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca75f98-79ba-43f9-96be-7de3f10edb61 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:33.620270 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca75f98-79ba-43f9-96be-7de3f10edb61 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:33.620539 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca75f98-79ba-43f9-96be-7de3f10edb61 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:33.627888 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca75f98-79ba-43f9-96be-7de3f10edb61 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:33.629384 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca75f98-79ba-43f9-96be-7de3f10edb61 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:33.641198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-0ca75f98-79ba-43f9-96be-7de3f10edb61 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:28:33.641198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:33.641198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:33.641198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:33.641198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:33.641198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:33.641198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:33.641198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:33.641198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:33.641198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:33.641198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:33.641198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:33.641198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:28:33.641198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:28:33.641198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:28:33.642268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:28:33.642268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:33.642268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:33.642268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:33.642268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:33.642268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:33.642268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:33.642268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:33.642268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:33.642268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:33.642268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:33.642268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:33.642268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:33.642268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:33.642268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:33.642268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:33.642268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:33.643563 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:28:33.643563 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:28:33.643563 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:28:33.643563 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:28:33.643563 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:28:33.643563 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:28:33.643563 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:28:33.643563 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:33.643563 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 535/1079] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:28:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:33.647958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca75f98-79ba-43f9-96be-7de3f10edb61 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:33.714200 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b396a293-ae80-48a4-85ae-e65786c2e69d None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:33.747444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15662c46-6d55-48e0-ad36-b10f4c12b39f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:33.748035 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca75f98-79ba-43f9-96be-7de3f10edb61 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:33.748144 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15662c46-6d55-48e0-ad36-b10f4c12b39f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:33.748336 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15662c46-6d55-48e0-ad36-b10f4c12b39f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:33.748610 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15662c46-6d55-48e0-ad36-b10f4c12b39f None admin] PATH_INFO: `/v3/users/f0a9ae0ae9a04d16b7d7baf8d840838e` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:33.749729 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15662c46-6d55-48e0-ad36-b10f4c12b39f None admin] RBAC: Authorizing `identity:get_user(user_id=f0a9ae0ae9a04d16b7d7baf8d840838e)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:33.751212 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15662c46-6d55-48e0-ad36-b10f4c12b39f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:33.758579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 545/1080] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:28:33 2022] GET /identity/v3/users/f0a9ae0ae9a04d16b7d7baf8d840838e => generated 285 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:33.759358 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15662c46-6d55-48e0-ad36-b10f4c12b39f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:33.784164 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85ef50d-75d0-4dbe-afb3-b3fff8756c77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:33.784766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85ef50d-75d0-4dbe-afb3-b3fff8756c77 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:33.785045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85ef50d-75d0-4dbe-afb3-b3fff8756c77 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:33.785312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85ef50d-75d0-4dbe-afb3-b3fff8756c77 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:33.785986 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85ef50d-75d0-4dbe-afb3-b3fff8756c77 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:33.786288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85ef50d-75d0-4dbe-afb3-b3fff8756c77 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:33.787730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85ef50d-75d0-4dbe-afb3-b3fff8756c77 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:33.799160 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 536/1081] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:28:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:33.805083 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85ef50d-75d0-4dbe-afb3-b3fff8756c77 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:33.823174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:33.832170 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:33.856992 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15662c46-6d55-48e0-ad36-b10f4c12b39f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:33.890336 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4475fa6-0661-497f-a041-7a2b0ff52229 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:33.890983 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4475fa6-0661-497f-a041-7a2b0ff52229 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:33.891270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4475fa6-0661-497f-a041-7a2b0ff52229 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:33.891545 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4475fa6-0661-497f-a041-7a2b0ff52229 None admin] PATH_INFO: `/v3/projects/d44d638c1b00415c8d972b575a7bd2b7` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:33.892641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4475fa6-0661-497f-a041-7a2b0ff52229 None admin] RBAC: Authorizing `identity:get_project(project_id=d44d638c1b00415c8d972b575a7bd2b7)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:33.894114 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4475fa6-0661-497f-a041-7a2b0ff52229 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:33.895744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 546/1082] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:28:33 2022] GET /identity/v3/projects/d44d638c1b00415c8d972b575a7bd2b7 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:33.902203 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85ef50d-75d0-4dbe-afb3-b3fff8756c77 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:33.902363 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4475fa6-0661-497f-a041-7a2b0ff52229 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:33.936447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02189908-61b2-4183-a847-296cf5786c1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:33.937098 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02189908-61b2-4183-a847-296cf5786c1d None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:33.937375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02189908-61b2-4183-a847-296cf5786c1d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:33.937647 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02189908-61b2-4183-a847-296cf5786c1d None admin] PATH_INFO: `/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/7ff68fb87700423ab53c90520a6bd3b9/roles/fa71d6ff1b04452c98c5830f178feb1b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:33.939811 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02189908-61b2-4183-a847-296cf5786c1d None admin] RBAC: Authorizing `identity:create_grant(project_id=c3802ced1fce4997a016d41eca843ef8, user_id=7ff68fb87700423ab53c90520a6bd3b9, role_id=fa71d6ff1b04452c98c5830f178feb1b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:33.941231 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02189908-61b2-4183-a847-296cf5786c1d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:33.998961 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4475fa6-0661-497f-a041-7a2b0ff52229 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:34.000071 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 537/1083] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:28:33 2022] PUT /identity/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/7ff68fb87700423ab53c90520a6bd3b9/roles/fa71d6ff1b04452c98c5830f178feb1b => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:28:34.038080 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce80e49-36eb-42f1-8896-f03ce6e0cb2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:34.038793 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce80e49-36eb-42f1-8896-f03ce6e0cb2a None admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:34.039075 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce80e49-36eb-42f1-8896-f03ce6e0cb2a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:34.039349 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce80e49-36eb-42f1-8896-f03ce6e0cb2a None admin] PATH_INFO: `/v3/projects/d44d638c1b00415c8d972b575a7bd2b7/users/f0a9ae0ae9a04d16b7d7baf8d840838e/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:34.041308 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce80e49-36eb-42f1-8896-f03ce6e0cb2a None admin] RBAC: Authorizing `identity:create_grant(project_id=d44d638c1b00415c8d972b575a7bd2b7, user_id=f0a9ae0ae9a04d16b7d7baf8d840838e, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:34.042794 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce80e49-36eb-42f1-8896-f03ce6e0cb2a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:34.045236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:34.121930 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 547/1084] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:28:33 2022] PUT /identity/v3/projects/d44d638c1b00415c8d972b575a7bd2b7/users/f0a9ae0ae9a04d16b7d7baf8d840838e/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:28:36.123051 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6d889a-c44f-472b-a251-713cdc80fc4c None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:36.123594 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6d889a-c44f-472b-a251-713cdc80fc4c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:36.123594 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6d889a-c44f-472b-a251-713cdc80fc4c None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:36.124959 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 538/1085] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:36.131273 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fb4569-ce4c-4d46-aba6-7159fc8c1400 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:36.131829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fb4569-ce4c-4d46-aba6-7159fc8c1400 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:36.131936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fb4569-ce4c-4d46-aba6-7159fc8c1400 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:36.151240 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-23fb4569-ce4c-4d46-aba6-7159fc8c1400 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:36.152480 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-23fb4569-ce4c-4d46-aba6-7159fc8c1400 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:36.222523 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 548/1086] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:36.250861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf92142-a34f-436f-a06c-cd073998de8e None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:36.251148 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf92142-a34f-436f-a06c-cd073998de8e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:36.251424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf92142-a34f-436f-a06c-cd073998de8e None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:36.252785 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 539/1087] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:36.258167 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0de10ad-e216-4546-a622-d1ce82e320c1 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:36.258462 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0de10ad-e216-4546-a622-d1ce82e320c1 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:36.258769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0de10ad-e216-4546-a622-d1ce82e320c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:36.276976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-a0de10ad-e216-4546-a622-d1ce82e320c1 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:36.278159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a0de10ad-e216-4546-a622-d1ce82e320c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:36.342609 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 549/1088] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:36.489800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf92142-a34f-436f-a06c-cd073998de8e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:36.572569 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:36.587975 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf92142-a34f-436f-a06c-cd073998de8e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:36.609217 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0de10ad-e216-4546-a622-d1ce82e320c1 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:36.619800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2065cd-4069-49e3-9636-67ba74c17b2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:36.620434 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2065cd-4069-49e3-9636-67ba74c17b2e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:36.620717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2065cd-4069-49e3-9636-67ba74c17b2e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:36.620988 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2065cd-4069-49e3-9636-67ba74c17b2e None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:36.631752 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2065cd-4069-49e3-9636-67ba74c17b2e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:28:36.637052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-7b2065cd-4069-49e3-9636-67ba74c17b2e None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 30 09:28:36.637052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:36.637052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:36.637052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:36.637052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:36.637052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:36.637052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:36.637052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:36.637052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:36.637052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:36.637052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:36.637052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:36.637052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:28:36.637052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:28:36.637052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:28:36.637052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:28:36.638011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:28:36.638011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:28:36.638011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:28:36.638011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:28:36.638011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:36.638011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:36.638011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:36.638011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:36.638011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:36.638011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:36.638011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:36.638011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:36.638011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:36.638011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:36.638011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:36.638011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:36.638968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:36.638968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:36.638968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:36.638968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:36.638968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:28:36.638968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:28:36.638968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:28:36.638968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:28:36.638968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:28:36.638968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:28:36.638968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 30 09:28:36.638968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:36.638968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 540/1089] 10.222.0.39 () {56 vars in 1120 bytes} [Fri Sep 30 09:28:36 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:36.644645 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2065cd-4069-49e3-9636-67ba74c17b2e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:36.663768 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:36.707797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0de10ad-e216-4546-a622-d1ce82e320c1 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:36.742480 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1af4e1f-d056-4aac-ae68-3c3abce06565 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:36.743063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1af4e1f-d056-4aac-ae68-3c3abce06565 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:36.743343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1af4e1f-d056-4aac-ae68-3c3abce06565 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:36.743461 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2065cd-4069-49e3-9636-67ba74c17b2e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:36.743616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1af4e1f-d056-4aac-ae68-3c3abce06565 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:36.753575 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1af4e1f-d056-4aac-ae68-3c3abce06565 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:28:36.759152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-c1af4e1f-d056-4aac-ae68-3c3abce06565 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:28:36.759152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:36.759152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:36.759152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:36.759152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:36.759152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:36.759152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:36.759152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:36.759152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:36.759152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:36.759152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:36.759152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:36.759152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:28:36.759152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:28:36.759152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:28:36.759152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:28:36.760315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:28:36.760315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:28:36.760315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:28:36.760315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:28:36.760315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:36.760315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:36.760315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:36.760315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:36.760315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:36.760315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:36.760315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:36.760315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:36.760315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:36.760315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:36.760315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:36.760315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:36.761740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:36.761740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:36.761740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:36.761740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:36.761740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:28:36.761740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:28:36.761740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:28:36.761740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:28:36.761740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:28:36.761740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:28:36.761740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:28:36.761740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:36.761740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 550/1090] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:28:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:36.766085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1af4e1f-d056-4aac-ae68-3c3abce06565 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:36.782268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd770b89-07c0-492c-a47e-b2d7f7b86518 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:36.782906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd770b89-07c0-492c-a47e-b2d7f7b86518 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:36.783188 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd770b89-07c0-492c-a47e-b2d7f7b86518 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:36.783470 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd770b89-07c0-492c-a47e-b2d7f7b86518 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:36.784124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd770b89-07c0-492c-a47e-b2d7f7b86518 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:36.784427 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd770b89-07c0-492c-a47e-b2d7f7b86518 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:36.785795 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd770b89-07c0-492c-a47e-b2d7f7b86518 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:36.793846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 541/1091] 10.222.0.39 () {56 vars in 1115 bytes} [Fri Sep 30 09:28:36 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:36.799487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd770b89-07c0-492c-a47e-b2d7f7b86518 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:36.810427 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:36.865405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1af4e1f-d056-4aac-ae68-3c3abce06565 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:36.879028 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:36.899545 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bf8882-5774-40fd-b8fb-7ab060909bed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:36.900131 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd770b89-07c0-492c-a47e-b2d7f7b86518 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:36.900271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bf8882-5774-40fd-b8fb-7ab060909bed None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:36.900409 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bf8882-5774-40fd-b8fb-7ab060909bed None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:36.900682 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bf8882-5774-40fd-b8fb-7ab060909bed None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:36.901377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bf8882-5774-40fd-b8fb-7ab060909bed None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:36.901644 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bf8882-5774-40fd-b8fb-7ab060909bed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:36.903041 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bf8882-5774-40fd-b8fb-7ab060909bed None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:36.912101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 551/1092] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:28:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:36.917991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bf8882-5774-40fd-b8fb-7ab060909bed None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:36.933355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad80ec8-988a-40b9-9a4a-422d03093238 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:36.933983 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad80ec8-988a-40b9-9a4a-422d03093238 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:36.934264 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad80ec8-988a-40b9-9a4a-422d03093238 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:36.934540 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad80ec8-988a-40b9-9a4a-422d03093238 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:36.944705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad80ec8-988a-40b9-9a4a-422d03093238 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:36.946734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad80ec8-988a-40b9-9a4a-422d03093238 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:36.960458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-8ad80ec8-988a-40b9-9a4a-422d03093238 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:36.960458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:36.960458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:36.960458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:36.960458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:36.960458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:36.960458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:36.960458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:36.960458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:36.960458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:36.960458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:36.960458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:36.960458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:28:36.960458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:28:36.960458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:28:36.960458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:28:36.961612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:36.961612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:36.961612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:36.961612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:36.961612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:36.961612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:36.961612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:36.961612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:36.961612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:36.961612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:36.961612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:36.961612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:36.961612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:36.961612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:36.961612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:36.961612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:36.961612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:28:36.962726 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:28:36.962726 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:36.962726 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:36.962726 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 542/1093] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:36.967466 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad80ec8-988a-40b9-9a4a-422d03093238 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:37.015893 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bf8882-5774-40fd-b8fb-7ab060909bed None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:37.025879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:37.049558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24718a0c-1eb4-4f6b-a855-9558b02de5b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:37.050183 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24718a0c-1eb4-4f6b-a855-9558b02de5b4 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:37.050461 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24718a0c-1eb4-4f6b-a855-9558b02de5b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:37.050768 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24718a0c-1eb4-4f6b-a855-9558b02de5b4 None admin] PATH_INFO: `/v3/users/f0a9ae0ae9a04d16b7d7baf8d840838e` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:37.051873 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24718a0c-1eb4-4f6b-a855-9558b02de5b4 None admin] RBAC: Authorizing `identity:get_user(user_id=f0a9ae0ae9a04d16b7d7baf8d840838e)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:37.053308 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24718a0c-1eb4-4f6b-a855-9558b02de5b4 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:37.055626 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 552/1094] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:28:36 2022] GET /identity/v3/users/f0a9ae0ae9a04d16b7d7baf8d840838e => generated 285 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:37.061281 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24718a0c-1eb4-4f6b-a855-9558b02de5b4 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:37.066000 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad80ec8-988a-40b9-9a4a-422d03093238 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:37.103403 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c601b5c-a606-4ff2-bcdf-55064e13763a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:37.104010 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c601b5c-a606-4ff2-bcdf-55064e13763a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:37.104293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c601b5c-a606-4ff2-bcdf-55064e13763a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:37.104570 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c601b5c-a606-4ff2-bcdf-55064e13763a None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:37.105202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c601b5c-a606-4ff2-bcdf-55064e13763a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:37.105508 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c601b5c-a606-4ff2-bcdf-55064e13763a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:37.106633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:37.106908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c601b5c-a606-4ff2-bcdf-55064e13763a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:37.119515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 543/1095] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:28:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:37.125079 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c601b5c-a606-4ff2-bcdf-55064e13763a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:37.161757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24718a0c-1eb4-4f6b-a855-9558b02de5b4 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:37.202798 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e243466f-db71-403b-a5de-1ebbb9a4fb4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:37.203415 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e243466f-db71-403b-a5de-1ebbb9a4fb4b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:37.203697 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e243466f-db71-403b-a5de-1ebbb9a4fb4b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:37.203974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e243466f-db71-403b-a5de-1ebbb9a4fb4b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:37.224187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c601b5c-a606-4ff2-bcdf-55064e13763a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:37.244029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:37.258502 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea36a84-620e-4c80-8a6f-df861299d33d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:37.259151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea36a84-620e-4c80-8a6f-df861299d33d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:37.259430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea36a84-620e-4c80-8a6f-df861299d33d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:37.259694 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea36a84-620e-4c80-8a6f-df861299d33d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:37.266526 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea36a84-620e-4c80-8a6f-df861299d33d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:37.267931 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea36a84-620e-4c80-8a6f-df861299d33d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:37.272123 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e243466f-db71-403b-a5de-1ebbb9a4fb4b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:37.273595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e243466f-db71-403b-a5de-1ebbb9a4fb4b None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:37.279114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-9ea36a84-620e-4c80-8a6f-df861299d33d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 30 09:28:37.279114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:37.279114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:37.279114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:37.279114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:37.279114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:37.279114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:37.279114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:37.279114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:37.279114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:37.279114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:37.279114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:37.279114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:28:37.279114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:28:37.279114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:28:37.279114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:28:37.280241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:37.280241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:37.280241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:28:37.280241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:28:37.280241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:28:37.280241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:28:37.280241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:37.280241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:37.280241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:37.280241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:37.280241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:37.280241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:37.280241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:37.280241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:37.280241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:37.280241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:37.280241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:37.280241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:37.281223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:37.281223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:37.281223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:28:37.281223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:28:37.281223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:28:37.281223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:28:37.281223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:28:37.281223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:28:37.281223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 30 09:28:37.281223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:37.281223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 544/1096] 10.222.0.39 () {56 vars in 1093 bytes} [Fri Sep 30 09:28:37 2022] GET /identity/v3/users/nova => generated 82 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:28:37.285966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea36a84-620e-4c80-8a6f-df861299d33d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:37.323436 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:37.376107 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 553/1097] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:28:37 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 30 09:28:37.382296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e243466f-db71-403b-a5de-1ebbb9a4fb4b None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:37.480242 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea36a84-620e-4c80-8a6f-df861299d33d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:37.511301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e243466f-db71-403b-a5de-1ebbb9a4fb4b None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:37.517933 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577ff3c7-45b5-470f-9ca3-e8ae4be9130c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:37.518477 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577ff3c7-45b5-470f-9ca3-e8ae4be9130c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:37.518788 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577ff3c7-45b5-470f-9ca3-e8ae4be9130c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:37.519057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577ff3c7-45b5-470f-9ca3-e8ae4be9130c None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:37.519767 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577ff3c7-45b5-470f-9ca3-e8ae4be9130c None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:37.520069 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577ff3c7-45b5-470f-9ca3-e8ae4be9130c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:37.521495 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577ff3c7-45b5-470f-9ca3-e8ae4be9130c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:37.538258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 545/1098] 10.222.0.39 () {56 vars in 1133 bytes} [Fri Sep 30 09:28:37 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:37.544538 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577ff3c7-45b5-470f-9ca3-e8ae4be9130c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:37.545532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebf0dc1-cd68-43e6-aae5-aabb6c7fa2b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:37.546154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebf0dc1-cd68-43e6-aae5-aabb6c7fa2b7 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:37.546432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebf0dc1-cd68-43e6-aae5-aabb6c7fa2b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:37.546749 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebf0dc1-cd68-43e6-aae5-aabb6c7fa2b7 None admin] PATH_INFO: `/v3/projects/d44d638c1b00415c8d972b575a7bd2b7` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:37.547841 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebf0dc1-cd68-43e6-aae5-aabb6c7fa2b7 None admin] RBAC: Authorizing `identity:get_project(project_id=d44d638c1b00415c8d972b575a7bd2b7)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:37.549277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebf0dc1-cd68-43e6-aae5-aabb6c7fa2b7 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:37.556702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 554/1099] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:28:37 2022] GET /identity/v3/projects/d44d638c1b00415c8d972b575a7bd2b7 => generated 311 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:37.557556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebf0dc1-cd68-43e6-aae5-aabb6c7fa2b7 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:37.572642 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:37.584715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:37.641645 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577ff3c7-45b5-470f-9ca3-e8ae4be9130c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:37.654832 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebf0dc1-cd68-43e6-aae5-aabb6c7fa2b7 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:37.675876 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba45623b-e121-448e-bd1e-22bfb396ac68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:37.676482 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba45623b-e121-448e-bd1e-22bfb396ac68 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:37.676761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba45623b-e121-448e-bd1e-22bfb396ac68 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:37.677037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba45623b-e121-448e-bd1e-22bfb396ac68 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:37.688697 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b65703-26a4-4faf-a495-8ab5f9c7461b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:37.689284 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b65703-26a4-4faf-a495-8ab5f9c7461b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:37.689564 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b65703-26a4-4faf-a495-8ab5f9c7461b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:37.689831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b65703-26a4-4faf-a495-8ab5f9c7461b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:37.690535 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b65703-26a4-4faf-a495-8ab5f9c7461b None admin] RBAC: Adding query filter params (role.id=e874c311084c4c9fb7691bce461d319b, scope.project.id=d44d638c1b00415c8d972b575a7bd2b7, user.id=f0a9ae0ae9a04d16b7d7baf8d840838e) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:37.690866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b65703-26a4-4faf-a495-8ab5f9c7461b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:37.692351 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b65703-26a4-4faf-a495-8ab5f9c7461b None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:37.716221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 555/1100] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:28:37 2022] GET /identity/v3/role_assignments?user.id=f0a9ae0ae9a04d16b7d7baf8d840838e&scope.project.id=d44d638c1b00415c8d972b575a7bd2b7&role.id=e874c311084c4c9fb7691bce461d319b => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:28:37.746814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba45623b-e121-448e-bd1e-22bfb396ac68 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:37.748325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba45623b-e121-448e-bd1e-22bfb396ac68 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:37.790735 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:37.858540 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 546/1101] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:28:37 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 30 09:28:37.859777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b65703-26a4-4faf-a495-8ab5f9c7461b None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:37.939597 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:37.962106 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b65703-26a4-4faf-a495-8ab5f9c7461b None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:37.995338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bcc095-f343-441c-a0a6-0888a5b992de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:37.996014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bcc095-f343-441c-a0a6-0888a5b992de None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:37.996292 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bcc095-f343-441c-a0a6-0888a5b992de None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:37.996558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bcc095-f343-441c-a0a6-0888a5b992de None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:38.007137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56bcc095-f343-441c-a0a6-0888a5b992de None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:38.008572 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56bcc095-f343-441c-a0a6-0888a5b992de None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:38.022163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-56bcc095-f343-441c-a0a6-0888a5b992de None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:38.022163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:38.022163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:38.022163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:38.022163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:38.022163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:38.022163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:38.022163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:38.022163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:38.022163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:38.022163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:38.022163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:38.022163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:28:38.022163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:28:38.022163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:28:38.022163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:28:38.023809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:38.023809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:38.023809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:38.023809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:38.023809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:38.023809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:38.023809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:38.023809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:38.023809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:38.023809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:38.023809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:38.023809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:38.023809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:38.023809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:38.023809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:38.023809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:38.023809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:28:38.024763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:28:38.024763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:28:38.024763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:38.024763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 556/1102] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:38.029394 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba45623b-e121-448e-bd1e-22bfb396ac68 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:38.131399 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba45623b-e121-448e-bd1e-22bfb396ac68 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:38.164336 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a2c511-ed36-41fe-aa21-a6c84a2da5e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:38.164934 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a2c511-ed36-41fe-aa21-a6c84a2da5e5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:38.165213 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a2c511-ed36-41fe-aa21-a6c84a2da5e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:38.165487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a2c511-ed36-41fe-aa21-a6c84a2da5e5 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:38.166110 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a2c511-ed36-41fe-aa21-a6c84a2da5e5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:38.166412 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a2c511-ed36-41fe-aa21-a6c84a2da5e5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:38.167798 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a2c511-ed36-41fe-aa21-a6c84a2da5e5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:38.174504 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:38.181948 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 547/1103] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:28:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:38.187544 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bcc095-f343-441c-a0a6-0888a5b992de None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:38.286254 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bcc095-f343-441c-a0a6-0888a5b992de None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:38.310898 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:38.318364 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e76c3e5-c087-445d-af54-0c958b260ba0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:38.319017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e76c3e5-c087-445d-af54-0c958b260ba0 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:38.319295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e76c3e5-c087-445d-af54-0c958b260ba0 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:38.319566 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e76c3e5-c087-445d-af54-0c958b260ba0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:38.326473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e76c3e5-c087-445d-af54-0c958b260ba0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:38.327950 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e76c3e5-c087-445d-af54-0c958b260ba0 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:38.339511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-2e76c3e5-c087-445d-af54-0c958b260ba0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:28:38.339511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:38.339511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:38.339511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:38.339511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:38.339511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:38.339511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:38.339511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:38.339511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:38.339511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:38.339511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:38.339511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:38.339511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:28:38.339511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:28:38.339511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:28:38.340444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:28:38.340444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:38.340444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:38.340444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:38.340444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:38.340444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:38.340444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:38.340444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:38.340444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:38.340444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:38.340444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:38.340444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:38.340444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:38.340444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:38.340444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:38.340444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:38.340444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:38.341391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:28:38.341391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:28:38.341391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:28:38.341391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:28:38.341391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:28:38.341391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:28:38.341391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:28:38.341391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:28:38.342267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 557/1104] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:28:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:38.346473 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a2c511-ed36-41fe-aa21-a6c84a2da5e5 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:38.441190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a2c511-ed36-41fe-aa21-a6c84a2da5e5 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:38.474051 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075ee96b-446e-44d1-92f6-5fd7940815ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:38.474683 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075ee96b-446e-44d1-92f6-5fd7940815ee None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:38.474961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075ee96b-446e-44d1-92f6-5fd7940815ee None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:38.475228 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075ee96b-446e-44d1-92f6-5fd7940815ee None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:38.475897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075ee96b-446e-44d1-92f6-5fd7940815ee None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:38.476196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075ee96b-446e-44d1-92f6-5fd7940815ee None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:38.477648 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075ee96b-446e-44d1-92f6-5fd7940815ee None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:38.488772 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 548/1105] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:28:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:38.495243 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e76c3e5-c087-445d-af54-0c958b260ba0 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:38.591586 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e76c3e5-c087-445d-af54-0c958b260ba0 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:38.625849 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ad7302-caa6-416a-9487-aedd4a4fa078 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:38.626448 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ad7302-caa6-416a-9487-aedd4a4fa078 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:38.626751 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ad7302-caa6-416a-9487-aedd4a4fa078 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:38.627022 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ad7302-caa6-416a-9487-aedd4a4fa078 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:38.627734 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ad7302-caa6-416a-9487-aedd4a4fa078 None admin] RBAC: Adding query filter params (role.id=fa71d6ff1b04452c98c5830f178feb1b, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=7ff68fb87700423ab53c90520a6bd3b9) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:38.628040 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ad7302-caa6-416a-9487-aedd4a4fa078 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:38.629468 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ad7302-caa6-416a-9487-aedd4a4fa078 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:38.635987 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 558/1106] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:28:38 2022] GET /identity/v3/role_assignments?user.id=7ff68fb87700423ab53c90520a6bd3b9&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=fa71d6ff1b04452c98c5830f178feb1b => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:39.871171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ad612b-ef7b-469a-96e7-ec9159a93a92 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:39.871771 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ad612b-ef7b-469a-96e7-ec9159a93a92 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:39.871771 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ad612b-ef7b-469a-96e7-ec9159a93a92 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:39.873037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 549/1107] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:39.878105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e7a5d8-0da1-4325-bb95-5a355039e784 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:39.878531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e7a5d8-0da1-4325-bb95-5a355039e784 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:39.878662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e7a5d8-0da1-4325-bb95-5a355039e784 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:39.897113 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-57e7a5d8-0da1-4325-bb95-5a355039e784 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:39.898929 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-57e7a5d8-0da1-4325-bb95-5a355039e784 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:39.944531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:39.963625 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 559/1108] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:40.231351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ad612b-ef7b-469a-96e7-ec9159a93a92 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:40.288623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:40.329593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ad612b-ef7b-469a-96e7-ec9159a93a92 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:40.360975 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572853ab-212f-4a71-ac86-b6b4b9fc93df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:40.361579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572853ab-212f-4a71-ac86-b6b4b9fc93df None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:40.361855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572853ab-212f-4a71-ac86-b6b4b9fc93df None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:40.362121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572853ab-212f-4a71-ac86-b6b4b9fc93df None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:40.363218 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572853ab-212f-4a71-ac86-b6b4b9fc93df None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:40.364640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572853ab-212f-4a71-ac86-b6b4b9fc93df None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:40.366158 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 550/1109] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:40 2022] GET /identity/v3/domains/default => generated 199 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:40.372280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e7a5d8-0da1-4325-bb95-5a355039e784 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:40.467440 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e7a5d8-0da1-4325-bb95-5a355039e784 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:40.500357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18ec130-a3eb-49e5-93d0-255599bcd71a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:40.500946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18ec130-a3eb-49e5-93d0-255599bcd71a None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:40.501226 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18ec130-a3eb-49e5-93d0-255599bcd71a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:40.501497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18ec130-a3eb-49e5-93d0-255599bcd71a None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:40.502134 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18ec130-a3eb-49e5-93d0-255599bcd71a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:40.503584 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18ec130-a3eb-49e5-93d0-255599bcd71a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:40.586175 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 560/1110] 10.222.0.39 () {60 vars in 1132 bytes} [Fri Sep 30 09:28:40 2022] POST /identity/v3/users => generated 286 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:28:42.733734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d0fb1b-45a4-40c6-acd6-6b91ba2b4bbd None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:42.734151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d0fb1b-45a4-40c6-acd6-6b91ba2b4bbd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:42.734278 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d0fb1b-45a4-40c6-acd6-6b91ba2b4bbd None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:42.735653 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 551/1111] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:42.740878 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3001289f-d663-4c29-8403-f7e019693835 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:42.741266 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3001289f-d663-4c29-8403-f7e019693835 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:42.741419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3001289f-d663-4c29-8403-f7e019693835 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:42.759829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-3001289f-d663-4c29-8403-f7e019693835 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:42.761001 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-3001289f-d663-4c29-8403-f7e019693835 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:42.793821 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:42.829448 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 561/1112] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:43.097463 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d0fb1b-45a4-40c6-acd6-6b91ba2b4bbd None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:43.200156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d0fb1b-45a4-40c6-acd6-6b91ba2b4bbd None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:43.224663 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:43.232567 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebf604c-d2ef-4076-b7db-dcbf96288993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:43.233190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebf604c-d2ef-4076-b7db-dcbf96288993 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:43.233470 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebf604c-d2ef-4076-b7db-dcbf96288993 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:43.233742 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebf604c-d2ef-4076-b7db-dcbf96288993 None admin] PATH_INFO: `/v3/roles/da6797f7fa24429095283b460be5c792` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:43.235133 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebf604c-d2ef-4076-b7db-dcbf96288993 None admin] RBAC: Authorizing `identity:get_role(role_id=da6797f7fa24429095283b460be5c792)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:43.236515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebf604c-d2ef-4076-b7db-dcbf96288993 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:43.237819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 552/1113] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:28:43 2022] GET /identity/v3/roles/da6797f7fa24429095283b460be5c792 => generated 223 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:43.244093 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3001289f-d663-4c29-8403-f7e019693835 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:43.340077 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3001289f-d663-4c29-8403-f7e019693835 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:43.373204 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e169c9e-ab7f-45ca-a69b-92245c50e0cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:43.373749 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e169c9e-ab7f-45ca-a69b-92245c50e0cc None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:43.374029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e169c9e-ab7f-45ca-a69b-92245c50e0cc None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:43.374302 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e169c9e-ab7f-45ca-a69b-92245c50e0cc None admin] PATH_INFO: `/v3/users/a1208e1cddb74cab9188ccdae2ee03cf` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:43.387180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e169c9e-ab7f-45ca-a69b-92245c50e0cc None admin] RBAC: Authorizing `identity:get_user(user_id=a1208e1cddb74cab9188ccdae2ee03cf)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:43.388700 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e169c9e-ab7f-45ca-a69b-92245c50e0cc None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:43.390262 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 562/1114] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:28:43 2022] GET /identity/v3/users/a1208e1cddb74cab9188ccdae2ee03cf => generated 286 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:43.396012 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebf604c-d2ef-4076-b7db-dcbf96288993 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:43.490772 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebf604c-d2ef-4076-b7db-dcbf96288993 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:43.522929 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3058dff8-98f9-41c7-98e6-2782359862a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:43.523523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3058dff8-98f9-41c7-98e6-2782359862a1 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:43.523802 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3058dff8-98f9-41c7-98e6-2782359862a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:43.524074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3058dff8-98f9-41c7-98e6-2782359862a1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:43.580668 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:43.587501 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3058dff8-98f9-41c7-98e6-2782359862a1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:43.588927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3058dff8-98f9-41c7-98e6-2782359862a1 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:43.682148 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 553/1115] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:28:43 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 286 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 30 09:28:43.687977 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e169c9e-ab7f-45ca-a69b-92245c50e0cc None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:43.714520 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:43.784497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e169c9e-ab7f-45ca-a69b-92245c50e0cc None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:43.817429 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e259e56-f323-497b-8ae3-fea98cf196a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:43.818050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e259e56-f323-497b-8ae3-fea98cf196a6 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:43.818332 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e259e56-f323-497b-8ae3-fea98cf196a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:43.818631 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e259e56-f323-497b-8ae3-fea98cf196a6 None admin] PATH_INFO: `/v3/projects/d44d638c1b00415c8d972b575a7bd2b7` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:43.819755 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e259e56-f323-497b-8ae3-fea98cf196a6 None admin] RBAC: Authorizing `identity:get_project(project_id=d44d638c1b00415c8d972b575a7bd2b7)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:43.821194 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e259e56-f323-497b-8ae3-fea98cf196a6 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:43.822823 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 563/1116] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:28:43 2022] GET /identity/v3/projects/d44d638c1b00415c8d972b575a7bd2b7 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:43.828865 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3058dff8-98f9-41c7-98e6-2782359862a1 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:43.918418 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:43.924914 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3058dff8-98f9-41c7-98e6-2782359862a1 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:43.957155 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00731d14-440a-41c1-a712-919a6809831d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:43.957725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00731d14-440a-41c1-a712-919a6809831d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:43.958002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00731d14-440a-41c1-a712-919a6809831d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:43.958274 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00731d14-440a-41c1-a712-919a6809831d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:43.959013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00731d14-440a-41c1-a712-919a6809831d None admin] RBAC: Adding query filter params (role.id=da6797f7fa24429095283b460be5c792, scope.project.id=d44d638c1b00415c8d972b575a7bd2b7, user.id=a1208e1cddb74cab9188ccdae2ee03cf) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:43.959323 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00731d14-440a-41c1-a712-919a6809831d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:43.961258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00731d14-440a-41c1-a712-919a6809831d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:43.966850 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 554/1117] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:28:43 2022] GET /identity/v3/role_assignments?user.id=a1208e1cddb74cab9188ccdae2ee03cf&scope.project.id=d44d638c1b00415c8d972b575a7bd2b7&role.id=da6797f7fa24429095283b460be5c792 => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:46.051163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f44346-a0a4-486a-a5c0-f99863500262 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:46.051712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f44346-a0a4-486a-a5c0-f99863500262 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:46.051712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f44346-a0a4-486a-a5c0-f99863500262 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:46.053072 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 564/1118] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:46.058641 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a8c1dc-cbde-4812-a5b7-8cddc27ce455 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:46.059799 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a8c1dc-cbde-4812-a5b7-8cddc27ce455 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:46.059799 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a8c1dc-cbde-4812-a5b7-8cddc27ce455 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:46.077916 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e1a8c1dc-cbde-4812-a5b7-8cddc27ce455 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:46.079120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e1a8c1dc-cbde-4812-a5b7-8cddc27ce455 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:46.143044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 555/1119] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:46 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:46.409008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f44346-a0a4-486a-a5c0-f99863500262 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:46.502318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:46.509952 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f44346-a0a4-486a-a5c0-f99863500262 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:46.542406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1a36cf-a90b-43f3-8a1a-b114bfecc0ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:46.543084 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1a36cf-a90b-43f3-8a1a-b114bfecc0ee None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:46.543365 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1a36cf-a90b-43f3-8a1a-b114bfecc0ee None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:46.543639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1a36cf-a90b-43f3-8a1a-b114bfecc0ee None admin] PATH_INFO: `/v3/roles/da6797f7fa24429095283b460be5c792` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:46.545003 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d1a36cf-a90b-43f3-8a1a-b114bfecc0ee None admin] RBAC: Authorizing `identity:get_role(role_id=da6797f7fa24429095283b460be5c792)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:46.546383 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d1a36cf-a90b-43f3-8a1a-b114bfecc0ee None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:46.547719 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 565/1120] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:28:46 2022] GET /identity/v3/roles/da6797f7fa24429095283b460be5c792 => generated 223 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:46.553479 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a8c1dc-cbde-4812-a5b7-8cddc27ce455 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:46.636537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:46.647823 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a8c1dc-cbde-4812-a5b7-8cddc27ce455 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:46.680550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a5e4c9-366d-4802-bc6f-fdbf05de6cd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:46.681101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a5e4c9-366d-4802-bc6f-fdbf05de6cd1 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:46.681381 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a5e4c9-366d-4802-bc6f-fdbf05de6cd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:46.681649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a5e4c9-366d-4802-bc6f-fdbf05de6cd1 None admin] PATH_INFO: `/v3/users/a1208e1cddb74cab9188ccdae2ee03cf` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:46.682753 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a5e4c9-366d-4802-bc6f-fdbf05de6cd1 None admin] RBAC: Authorizing `identity:get_user(user_id=a1208e1cddb74cab9188ccdae2ee03cf)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:46.684154 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a5e4c9-366d-4802-bc6f-fdbf05de6cd1 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:46.685671 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 556/1121] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:28:46 2022] GET /identity/v3/users/a1208e1cddb74cab9188ccdae2ee03cf => generated 286 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:46.691261 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1a36cf-a90b-43f3-8a1a-b114bfecc0ee None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:46.787508 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1a36cf-a90b-43f3-8a1a-b114bfecc0ee None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:46.820272 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423c13db-7fbd-4e14-9533-d7a56882a154 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:46.820877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423c13db-7fbd-4e14-9533-d7a56882a154 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:46.821154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423c13db-7fbd-4e14-9533-d7a56882a154 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:46.821423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423c13db-7fbd-4e14-9533-d7a56882a154 None admin] PATH_INFO: `/v3/projects/d44d638c1b00415c8d972b575a7bd2b7` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:46.822509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423c13db-7fbd-4e14-9533-d7a56882a154 None admin] RBAC: Authorizing `identity:get_project(project_id=d44d638c1b00415c8d972b575a7bd2b7)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:46.823959 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423c13db-7fbd-4e14-9533-d7a56882a154 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:46.825538 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 566/1122] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:28:46 2022] GET /identity/v3/projects/d44d638c1b00415c8d972b575a7bd2b7 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:46.831366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a5e4c9-366d-4802-bc6f-fdbf05de6cd1 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:46.926069 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a5e4c9-366d-4802-bc6f-fdbf05de6cd1 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:46.957757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e308f51-944f-45fa-8bf1-c08e39dc3a9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:46.958338 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e308f51-944f-45fa-8bf1-c08e39dc3a9e None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:46.958640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e308f51-944f-45fa-8bf1-c08e39dc3a9e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:46.958919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e308f51-944f-45fa-8bf1-c08e39dc3a9e None admin] PATH_INFO: `/v3/projects/d44d638c1b00415c8d972b575a7bd2b7/users/a1208e1cddb74cab9188ccdae2ee03cf/roles/da6797f7fa24429095283b460be5c792` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:46.961505 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e308f51-944f-45fa-8bf1-c08e39dc3a9e None admin] RBAC: Authorizing `identity:create_grant(project_id=d44d638c1b00415c8d972b575a7bd2b7, user_id=a1208e1cddb74cab9188ccdae2ee03cf, role_id=da6797f7fa24429095283b460be5c792)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:46.962942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e308f51-944f-45fa-8bf1-c08e39dc3a9e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:47.033762 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 557/1123] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:28:46 2022] PUT /identity/v3/projects/d44d638c1b00415c8d972b575a7bd2b7/users/a1208e1cddb74cab9188ccdae2ee03cf/roles/da6797f7fa24429095283b460be5c792 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:28:49.128416 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb518dd2-bd2c-4e74-9c27-acf8624fc2c6 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:49.128960 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb518dd2-bd2c-4e74-9c27-acf8624fc2c6 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:49.128960 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb518dd2-bd2c-4e74-9c27-acf8624fc2c6 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:49.130537 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 567/1124] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:49.136479 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9c65f8-b10e-4efe-9249-72fb729f2f76 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:49.136928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9c65f8-b10e-4efe-9249-72fb729f2f76 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:49.137019 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9c65f8-b10e-4efe-9249-72fb729f2f76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:49.154714 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ef9c65f8-b10e-4efe-9249-72fb729f2f76 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:49.155882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ef9c65f8-b10e-4efe-9249-72fb729f2f76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:49.166005 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:49.220187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 558/1125] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 30 09:28:49.490335 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb518dd2-bd2c-4e74-9c27-acf8624fc2c6 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:49.520207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:49.588320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb518dd2-bd2c-4e74-9c27-acf8624fc2c6 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:49.621003 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cc8fe5-50f9-4c70-b56d-667ae1a66033 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:49.621608 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cc8fe5-50f9-4c70-b56d-667ae1a66033 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:49.621883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cc8fe5-50f9-4c70-b56d-667ae1a66033 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:49.622154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cc8fe5-50f9-4c70-b56d-667ae1a66033 None admin] PATH_INFO: `/v3/roles/da6797f7fa24429095283b460be5c792` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:49.623522 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cc8fe5-50f9-4c70-b56d-667ae1a66033 None admin] RBAC: Authorizing `identity:get_role(role_id=da6797f7fa24429095283b460be5c792)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:49.624894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cc8fe5-50f9-4c70-b56d-667ae1a66033 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:49.626174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 568/1126] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:28:49 2022] GET /identity/v3/roles/da6797f7fa24429095283b460be5c792 => generated 223 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:49.632263 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9c65f8-b10e-4efe-9249-72fb729f2f76 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:49.724632 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9c65f8-b10e-4efe-9249-72fb729f2f76 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:49.756739 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e419c72a-cd22-40e9-a8c4-6360977fd6a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:49.757320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e419c72a-cd22-40e9-a8c4-6360977fd6a8 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:49.757596 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e419c72a-cd22-40e9-a8c4-6360977fd6a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:49.757865 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e419c72a-cd22-40e9-a8c4-6360977fd6a8 None admin] PATH_INFO: `/v3/users/a1208e1cddb74cab9188ccdae2ee03cf` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:49.758985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e419c72a-cd22-40e9-a8c4-6360977fd6a8 None admin] RBAC: Authorizing `identity:get_user(user_id=a1208e1cddb74cab9188ccdae2ee03cf)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:49.760385 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e419c72a-cd22-40e9-a8c4-6360977fd6a8 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:49.761907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 559/1127] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:28:49 2022] GET /identity/v3/users/a1208e1cddb74cab9188ccdae2ee03cf => generated 286 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:49.767441 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cc8fe5-50f9-4c70-b56d-667ae1a66033 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:49.859278 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:49.862907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cc8fe5-50f9-4c70-b56d-667ae1a66033 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:49.895155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d4faf2-b6ec-476a-a326-61797b3ec8f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:49.895743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d4faf2-b6ec-476a-a326-61797b3ec8f7 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:49.896018 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d4faf2-b6ec-476a-a326-61797b3ec8f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:49.896285 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d4faf2-b6ec-476a-a326-61797b3ec8f7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:49.959641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d4faf2-b6ec-476a-a326-61797b3ec8f7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:49.961593 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d4faf2-b6ec-476a-a326-61797b3ec8f7 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:50.050544 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:50.054711 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 569/1128] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:28:49 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 288 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 30 09:28:50.060625 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e419c72a-cd22-40e9-a8c4-6360977fd6a8 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:50.077077 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:50.157698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e419c72a-cd22-40e9-a8c4-6360977fd6a8 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:50.189280 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f785d0c-4377-4565-91ce-582024b6b506 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:50.189904 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f785d0c-4377-4565-91ce-582024b6b506 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:50.190181 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f785d0c-4377-4565-91ce-582024b6b506 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:50.190450 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f785d0c-4377-4565-91ce-582024b6b506 None admin] PATH_INFO: `/v3/projects/d44d638c1b00415c8d972b575a7bd2b7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:50.191575 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f785d0c-4377-4565-91ce-582024b6b506 None admin] RBAC: Authorizing `identity:get_project(project_id=d44d638c1b00415c8d972b575a7bd2b7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:50.192997 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f785d0c-4377-4565-91ce-582024b6b506 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:50.195180 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 560/1129] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:28:50 2022] GET /identity/v3/projects/d44d638c1b00415c8d972b575a7bd2b7 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:50.201356 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d4faf2-b6ec-476a-a326-61797b3ec8f7 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:50.296866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d4faf2-b6ec-476a-a326-61797b3ec8f7 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:50.329614 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344ec7ac-9a4c-42cd-9d30-b36ec8ae0c63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:50.330142 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344ec7ac-9a4c-42cd-9d30-b36ec8ae0c63 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:50.330422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344ec7ac-9a4c-42cd-9d30-b36ec8ae0c63 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:50.330725 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344ec7ac-9a4c-42cd-9d30-b36ec8ae0c63 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:50.331434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344ec7ac-9a4c-42cd-9d30-b36ec8ae0c63 None admin] RBAC: Adding query filter params (role.id=da6797f7fa24429095283b460be5c792, scope.project.id=d44d638c1b00415c8d972b575a7bd2b7, user.id=a1208e1cddb74cab9188ccdae2ee03cf) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:50.331740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344ec7ac-9a4c-42cd-9d30-b36ec8ae0c63 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:50.333178 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344ec7ac-9a4c-42cd-9d30-b36ec8ae0c63 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:50.339335 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 570/1130] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:28:50 2022] GET /identity/v3/role_assignments?user.id=a1208e1cddb74cab9188ccdae2ee03cf&scope.project.id=d44d638c1b00415c8d972b575a7bd2b7&role.id=da6797f7fa24429095283b460be5c792 => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:52.461420 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a6ebf1-24a4-425d-aaa5-1c22fc70edc6 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:52.461939 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a6ebf1-24a4-425d-aaa5-1c22fc70edc6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:52.461939 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a6ebf1-24a4-425d-aaa5-1c22fc70edc6 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:52.463248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 561/1131] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:52.468468 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d50402-a5dd-4979-9c1e-97d082a439c1 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:52.468860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d50402-a5dd-4979-9c1e-97d082a439c1 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:52.469013 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d50402-a5dd-4979-9c1e-97d082a439c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:52.487234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-97d50402-a5dd-4979-9c1e-97d082a439c1 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:52.488403 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-97d50402-a5dd-4979-9c1e-97d082a439c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:52.542340 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:52.552253 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 571/1132] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:52.818127 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a6ebf1-24a4-425d-aaa5-1c22fc70edc6 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:52.897543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:52.915011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a6ebf1-24a4-425d-aaa5-1c22fc70edc6 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:52.948238 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dc155a-b1a1-485a-a6dd-33bad56ef971 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:52.948844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dc155a-b1a1-485a-a6dd-33bad56ef971 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:52.949118 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dc155a-b1a1-485a-a6dd-33bad56ef971 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:52.949387 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dc155a-b1a1-485a-a6dd-33bad56ef971 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:52.950465 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6dc155a-b1a1-485a-a6dd-33bad56ef971 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:52.951912 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6dc155a-b1a1-485a-a6dd-33bad56ef971 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:52.953424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 562/1133] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:52 2022] GET /identity/v3/domains/default => generated 199 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:52.959375 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d50402-a5dd-4979-9c1e-97d082a439c1 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:53.055076 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d50402-a5dd-4979-9c1e-97d082a439c1 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:53.088091 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7605ff7f-3d8a-4a55-aed2-73b6283adc3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:53.088623 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7605ff7f-3d8a-4a55-aed2-73b6283adc3a None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:53.088899 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7605ff7f-3d8a-4a55-aed2-73b6283adc3a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:53.089173 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7605ff7f-3d8a-4a55-aed2-73b6283adc3a None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:53.089861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7605ff7f-3d8a-4a55-aed2-73b6283adc3a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:53.091310 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7605ff7f-3d8a-4a55-aed2-73b6283adc3a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:53.184141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 572/1134] 10.222.0.39 () {60 vars in 1141 bytes} [Fri Sep 30 09:28:52 2022] POST /identity/v3/projects => generated 311 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:28:55.360021 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6df2f7-3399-47df-bad2-915f801737e1 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:55.360576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6df2f7-3399-47df-bad2-915f801737e1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:55.360576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6df2f7-3399-47df-bad2-915f801737e1 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:55.361874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 563/1135] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:55.367062 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f7d6c9-7153-4597-bdf1-61d51f6d9541 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:55.367454 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f7d6c9-7153-4597-bdf1-61d51f6d9541 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:55.367619 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f7d6c9-7153-4597-bdf1-61d51f6d9541 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:55.386122 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-52f7d6c9-7153-4597-bdf1-61d51f6d9541 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:55.387336 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-52f7d6c9-7153-4597-bdf1-61d51f6d9541 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:55.425964 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:55.451457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 573/1136] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:55.726123 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6df2f7-3399-47df-bad2-915f801737e1 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:55.823275 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6df2f7-3399-47df-bad2-915f801737e1 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:55.854395 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543d80fe-3c12-41d3-b60d-8b43abbc2c06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:55.855030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543d80fe-3c12-41d3-b60d-8b43abbc2c06 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:55.855306 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543d80fe-3c12-41d3-b60d-8b43abbc2c06 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:55.855573 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543d80fe-3c12-41d3-b60d-8b43abbc2c06 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:55.856649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543d80fe-3c12-41d3-b60d-8b43abbc2c06 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:55.858054 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543d80fe-3c12-41d3-b60d-8b43abbc2c06 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:55.859592 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 564/1137] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:28:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:55.865248 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f7d6c9-7153-4597-bdf1-61d51f6d9541 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:55.960827 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f7d6c9-7153-4597-bdf1-61d51f6d9541 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:55.993167 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f07ad8b-3230-495d-9594-685632693545 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:55.993698 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f07ad8b-3230-495d-9594-685632693545 None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:55.993978 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f07ad8b-3230-495d-9594-685632693545 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:55.994253 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f07ad8b-3230-495d-9594-685632693545 None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:55.994916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f07ad8b-3230-495d-9594-685632693545 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:55.996342 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f07ad8b-3230-495d-9594-685632693545 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:56.102206 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 574/1138] 10.222.0.39 () {60 vars in 1132 bytes} [Fri Sep 30 09:28:55 2022] POST /identity/v3/users => generated 286 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:28:58.268802 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ba87bd-086c-420f-9316-5993e7596ef2 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:58.269279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ba87bd-086c-420f-9316-5993e7596ef2 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:58.269368 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ba87bd-086c-420f-9316-5993e7596ef2 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:58.270834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 565/1139] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:28:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:28:58.276436 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db863f6c-4634-4ac9-a1dd-f1f61a94d33e None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:58.276858 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db863f6c-4634-4ac9-a1dd-f1f61a94d33e None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:58.276990 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db863f6c-4634-4ac9-a1dd-f1f61a94d33e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:58.295385 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-db863f6c-4634-4ac9-a1dd-f1f61a94d33e None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:28:58.296564 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-db863f6c-4634-4ac9-a1dd-f1f61a94d33e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:28:58.319742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:58.360656 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 575/1140] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:28:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:28:58.626160 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ba87bd-086c-420f-9316-5993e7596ef2 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:58.645184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:58.728256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ba87bd-086c-420f-9316-5993e7596ef2 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:58.761258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3999fd-e2f5-4d51-9e19-c4d9c3912880 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:58.761873 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3999fd-e2f5-4d51-9e19-c4d9c3912880 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:58.762152 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3999fd-e2f5-4d51-9e19-c4d9c3912880 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:58.762425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3999fd-e2f5-4d51-9e19-c4d9c3912880 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:58.772088 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3999fd-e2f5-4d51-9e19-c4d9c3912880 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:28:58.776810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-4f3999fd-e2f5-4d51-9e19-c4d9c3912880 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:28:58.776810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:28:58.776810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:28:58.776810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:28:58.776810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:28:58.776810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:28:58.776810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:28:58.776810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:28:58.776810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:28:58.776810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:28:58.776810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:28:58.776810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:28:58.776810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:28:58.776810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:28:58.776810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:28:58.776810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:28:58.778001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:28:58.778001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:28:58.778001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:28:58.778001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:28:58.778001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:28:58.778001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:28:58.778001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:28:58.778001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:28:58.778001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:28:58.778001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:28:58.778001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:28:58.778001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:28:58.778001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:28:58.778001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:28:58.778001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:28:58.778001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:28:58.779138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:28:58.779138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:28:58.779138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:28:58.779138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:28:58.779138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:28:58.779138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:28:58.779138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:28:58.779138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:28:58.779138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:28:58.779138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:28:58.779138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:28:58.779138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:28:58.779138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 566/1141] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:28:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:28:58.783963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db863f6c-4634-4ac9-a1dd-f1f61a94d33e None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:58.879492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db863f6c-4634-4ac9-a1dd-f1f61a94d33e None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:58.912205 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aec520-330a-45f0-88a8-541f4b82b1bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:58.912815 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aec520-330a-45f0-88a8-541f4b82b1bb None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:58.913093 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aec520-330a-45f0-88a8-541f4b82b1bb None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:58.913368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aec520-330a-45f0-88a8-541f4b82b1bb None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:58.914024 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1aec520-330a-45f0-88a8-541f4b82b1bb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:58.914324 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1aec520-330a-45f0-88a8-541f4b82b1bb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:58.915730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1aec520-330a-45f0-88a8-541f4b82b1bb None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:58.923728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 576/1142] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:28:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:58.930260 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3999fd-e2f5-4d51-9e19-c4d9c3912880 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:59.007074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:59.027738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3999fd-e2f5-4d51-9e19-c4d9c3912880 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:59.061081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85904c51-de22-4576-8f17-44d43dabfb9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:59.061689 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85904c51-de22-4576-8f17-44d43dabfb9b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:59.061971 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85904c51-de22-4576-8f17-44d43dabfb9b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:59.062246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85904c51-de22-4576-8f17-44d43dabfb9b None admin] PATH_INFO: `/v3/users/079da78e11b542b5b649433b1528a604` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:59.075509 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85904c51-de22-4576-8f17-44d43dabfb9b None admin] RBAC: Authorizing `identity:get_user(user_id=079da78e11b542b5b649433b1528a604)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:59.077009 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85904c51-de22-4576-8f17-44d43dabfb9b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:59.078576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 567/1143] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:28:58 2022] GET /identity/v3/users/079da78e11b542b5b649433b1528a604 => generated 286 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:59.084430 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aec520-330a-45f0-88a8-541f4b82b1bb None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:59.109466 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:59.181723 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aec520-330a-45f0-88a8-541f4b82b1bb None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:59.213905 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee1151b-7536-4cb1-a40b-b2fabf59f13b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:59.214512 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee1151b-7536-4cb1-a40b-b2fabf59f13b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:59.214826 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee1151b-7536-4cb1-a40b-b2fabf59f13b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:59.215099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee1151b-7536-4cb1-a40b-b2fabf59f13b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:59.279578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee1151b-7536-4cb1-a40b-b2fabf59f13b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:59.281031 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee1151b-7536-4cb1-a40b-b2fabf59f13b None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:59.303648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:59.374918 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 577/1144] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:28:59 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 30 09:28:59.381198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85904c51-de22-4576-8f17-44d43dabfb9b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:59.476338 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85904c51-de22-4576-8f17-44d43dabfb9b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:59.508246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e392aa-b223-45da-a4c3-114f37de824d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:59.508858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e392aa-b223-45da-a4c3-114f37de824d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:59.509135 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e392aa-b223-45da-a4c3-114f37de824d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:59.509404 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e392aa-b223-45da-a4c3-114f37de824d None admin] PATH_INFO: `/v3/projects/7c1dd6274b8a421faa6b4683e26815c3` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:59.510482 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e392aa-b223-45da-a4c3-114f37de824d None admin] RBAC: Authorizing `identity:get_project(project_id=7c1dd6274b8a421faa6b4683e26815c3)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:59.511929 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e392aa-b223-45da-a4c3-114f37de824d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:59.513489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 568/1145] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:28:59 2022] GET /identity/v3/projects/7c1dd6274b8a421faa6b4683e26815c3 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:28:59.519508 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee1151b-7536-4cb1-a40b-b2fabf59f13b None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:28:59.615082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee1151b-7536-4cb1-a40b-b2fabf59f13b None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:28:59.636481 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:28:59.647583 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd7ad0d-1619-42bc-a78e-9139f5ca3169 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:28:59.652459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd7ad0d-1619-42bc-a78e-9139f5ca3169 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:28:59.652459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd7ad0d-1619-42bc-a78e-9139f5ca3169 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:28:59.652459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd7ad0d-1619-42bc-a78e-9139f5ca3169 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:28:59.652459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd7ad0d-1619-42bc-a78e-9139f5ca3169 None admin] RBAC: Adding query filter params (role.id=e874c311084c4c9fb7691bce461d319b, scope.project.id=7c1dd6274b8a421faa6b4683e26815c3, user.id=079da78e11b542b5b649433b1528a604) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:28:59.652459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd7ad0d-1619-42bc-a78e-9139f5ca3169 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:28:59.652459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd7ad0d-1619-42bc-a78e-9139f5ca3169 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:28:59.656731 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 578/1146] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:28:59 2022] GET /identity/v3/role_assignments?user.id=079da78e11b542b5b649433b1528a604&scope.project.id=7c1dd6274b8a421faa6b4683e26815c3&role.id=e874c311084c4c9fb7691bce461d319b => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:01.741121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e5a395-cda2-4d0a-bffc-fbd76726ee59 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:01.742171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e5a395-cda2-4d0a-bffc-fbd76726ee59 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:01.742171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e5a395-cda2-4d0a-bffc-fbd76726ee59 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:01.743654 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 569/1147] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:29:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:29:01.748712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc20c71-a9d6-4257-922c-8cb54448c7c2 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:01.749098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc20c71-a9d6-4257-922c-8cb54448c7c2 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:01.749270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc20c71-a9d6-4257-922c-8cb54448c7c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:01.767860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-0dc20c71-a9d6-4257-922c-8cb54448c7c2 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:29:01.769042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0dc20c71-a9d6-4257-922c-8cb54448c7c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:29:01.832693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 579/1148] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:29:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:29:02.107942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e5a395-cda2-4d0a-bffc-fbd76726ee59 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:02.111689 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:02.205367 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e5a395-cda2-4d0a-bffc-fbd76726ee59 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:02.237254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2bd2ff-fa38-4362-91b4-55227d7263a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:02.237892 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2bd2ff-fa38-4362-91b4-55227d7263a8 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:02.238167 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2bd2ff-fa38-4362-91b4-55227d7263a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:02.238437 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2bd2ff-fa38-4362-91b4-55227d7263a8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:02.248649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2bd2ff-fa38-4362-91b4-55227d7263a8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:29:02.253312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-ad2bd2ff-fa38-4362-91b4-55227d7263a8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:29:02.253312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:29:02.253312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:29:02.253312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:29:02.253312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:29:02.253312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:29:02.253312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:29:02.253312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:29:02.253312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:29:02.253312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:29:02.253312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:29:02.253312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:29:02.253312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:29:02.253312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:29:02.253312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:29:02.253312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:29:02.254718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:29:02.254718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:29:02.254718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:29:02.254718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:29:02.254718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:29:02.254718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:29:02.254718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:29:02.254718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:29:02.254718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:29:02.254718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:29:02.254718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:29:02.254718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:29:02.254718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:29:02.254718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:29:02.254718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:29:02.254718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:29:02.255913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:29:02.255913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:29:02.255913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:29:02.255913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:29:02.255913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:29:02.255913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:29:02.255913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:29:02.255913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:29:02.255913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:29:02.255913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:29:02.255913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:29:02.255913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:29:02.255913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 570/1149] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:29:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:29:02.260738 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc20c71-a9d6-4257-922c-8cb54448c7c2 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:02.354900 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc20c71-a9d6-4257-922c-8cb54448c7c2 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:02.370829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:02.387627 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2c5ea8-fc36-4f9b-b90a-b79860c70860 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:02.388217 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2c5ea8-fc36-4f9b-b90a-b79860c70860 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:02.388493 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2c5ea8-fc36-4f9b-b90a-b79860c70860 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:02.388763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2c5ea8-fc36-4f9b-b90a-b79860c70860 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:02.389410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2c5ea8-fc36-4f9b-b90a-b79860c70860 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:29:02.389763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2c5ea8-fc36-4f9b-b90a-b79860c70860 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:02.391155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2c5ea8-fc36-4f9b-b90a-b79860c70860 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:02.399560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 580/1150] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:29:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:02.404878 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2bd2ff-fa38-4362-91b4-55227d7263a8 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:02.464353 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:02.500324 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2bd2ff-fa38-4362-91b4-55227d7263a8 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:02.531999 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e4a4c2-ce0d-4c42-a3d2-0f3e52df8316 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:02.532601 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e4a4c2-ce0d-4c42-a3d2-0f3e52df8316 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:02.532874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e4a4c2-ce0d-4c42-a3d2-0f3e52df8316 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:02.533141 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e4a4c2-ce0d-4c42-a3d2-0f3e52df8316 None admin] PATH_INFO: `/v3/users/079da78e11b542b5b649433b1528a604` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:02.534226 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e4a4c2-ce0d-4c42-a3d2-0f3e52df8316 None admin] RBAC: Authorizing `identity:get_user(user_id=079da78e11b542b5b649433b1528a604)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:02.535678 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e4a4c2-ce0d-4c42-a3d2-0f3e52df8316 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:02.537193 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 571/1151] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:29:02 2022] GET /identity/v3/users/079da78e11b542b5b649433b1528a604 => generated 286 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:02.543346 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2c5ea8-fc36-4f9b-b90a-b79860c70860 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:02.639653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2c5ea8-fc36-4f9b-b90a-b79860c70860 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:02.672404 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb261ad-2f48-4060-b141-0c0ee31e3b91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:02.673011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb261ad-2f48-4060-b141-0c0ee31e3b91 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:02.673288 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb261ad-2f48-4060-b141-0c0ee31e3b91 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:02.673559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb261ad-2f48-4060-b141-0c0ee31e3b91 None admin] PATH_INFO: `/v3/projects/7c1dd6274b8a421faa6b4683e26815c3` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:02.674672 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb261ad-2f48-4060-b141-0c0ee31e3b91 None admin] RBAC: Authorizing `identity:get_project(project_id=7c1dd6274b8a421faa6b4683e26815c3)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:02.676104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb261ad-2f48-4060-b141-0c0ee31e3b91 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:02.677673 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 581/1152] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:29:02 2022] GET /identity/v3/projects/7c1dd6274b8a421faa6b4683e26815c3 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:02.683346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e4a4c2-ce0d-4c42-a3d2-0f3e52df8316 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:02.777814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e4a4c2-ce0d-4c42-a3d2-0f3e52df8316 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:02.805200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:02.809904 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbf71ed-4145-4f0d-9531-d50a8912e239 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:02.810538 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbf71ed-4145-4f0d-9531-d50a8912e239 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:02.810845 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbf71ed-4145-4f0d-9531-d50a8912e239 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:02.811117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbf71ed-4145-4f0d-9531-d50a8912e239 None admin] PATH_INFO: `/v3/projects/7c1dd6274b8a421faa6b4683e26815c3/users/079da78e11b542b5b649433b1528a604/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:02.812948 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbf71ed-4145-4f0d-9531-d50a8912e239 None admin] RBAC: Authorizing `identity:create_grant(project_id=7c1dd6274b8a421faa6b4683e26815c3, user_id=079da78e11b542b5b649433b1528a604, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:02.814367 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbf71ed-4145-4f0d-9531-d50a8912e239 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:02.918447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 572/1153] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:29:02 2022] PUT /identity/v3/projects/7c1dd6274b8a421faa6b4683e26815c3/users/079da78e11b542b5b649433b1528a604/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:29:05.018749 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a625b5da-296e-41c8-900b-458298435fc7 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:05.019276 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a625b5da-296e-41c8-900b-458298435fc7 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:05.019276 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a625b5da-296e-41c8-900b-458298435fc7 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:05.020605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 582/1154] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:29:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:29:05.025915 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e00541c-1b09-46fc-b8ec-371a1e5f67dc None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:05.026308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e00541c-1b09-46fc-b8ec-371a1e5f67dc None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:05.026467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e00541c-1b09-46fc-b8ec-371a1e5f67dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:05.044867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9e00541c-1b09-46fc-b8ec-371a1e5f67dc None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:29:05.046046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9e00541c-1b09-46fc-b8ec-371a1e5f67dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:29:05.110657 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 573/1155] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:29:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:29:05.378550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a625b5da-296e-41c8-900b-458298435fc7 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:05.425368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:05.477768 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a625b5da-296e-41c8-900b-458298435fc7 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:05.510225 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d0c19d-3edb-4d82-ac01-0d365f207eb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:05.510893 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d0c19d-3edb-4d82-ac01-0d365f207eb9 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:05.511171 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d0c19d-3edb-4d82-ac01-0d365f207eb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:05.511443 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d0c19d-3edb-4d82-ac01-0d365f207eb9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:05.521185 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d0c19d-3edb-4d82-ac01-0d365f207eb9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:29:05.526526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-72d0c19d-3edb-4d82-ac01-0d365f207eb9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:29:05.526526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:29:05.526526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:29:05.526526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:29:05.526526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:29:05.526526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:29:05.526526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:29:05.526526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:29:05.526526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:29:05.526526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:29:05.526526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:29:05.526526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:29:05.526526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:29:05.526526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:29:05.526526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:29:05.526526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:29:05.527637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:29:05.527637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:29:05.527637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:29:05.527637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:29:05.527637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:29:05.527637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:29:05.527637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:29:05.527637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:29:05.527637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:29:05.527637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:29:05.527637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:29:05.527637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:29:05.527637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:29:05.527637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:29:05.527637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:29:05.527637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:29:05.528810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:29:05.528810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:29:05.528810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:29:05.528810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:29:05.528810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:29:05.528810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:29:05.528810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:29:05.528810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:29:05.528810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:29:05.528810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:29:05.528810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:29:05.528810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:29:05.528810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 583/1156] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:29:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:29:05.533528 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e00541c-1b09-46fc-b8ec-371a1e5f67dc None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:05.627683 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e00541c-1b09-46fc-b8ec-371a1e5f67dc None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:05.649823 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:05.660695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d00e02d-6bb9-4a1c-9bc4-76a6d22eb378 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:05.661844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d00e02d-6bb9-4a1c-9bc4-76a6d22eb378 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:05.661844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d00e02d-6bb9-4a1c-9bc4-76a6d22eb378 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:05.661844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d00e02d-6bb9-4a1c-9bc4-76a6d22eb378 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:05.662463 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d00e02d-6bb9-4a1c-9bc4-76a6d22eb378 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:29:05.662877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d00e02d-6bb9-4a1c-9bc4-76a6d22eb378 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:05.664848 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d00e02d-6bb9-4a1c-9bc4-76a6d22eb378 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:05.672794 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 574/1157] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:29:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:05.678138 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d0c19d-3edb-4d82-ac01-0d365f207eb9 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:05.773173 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d0c19d-3edb-4d82-ac01-0d365f207eb9 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:05.780151 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:05.805876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac60539-2345-4081-9e21-494be8a9f501 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:05.806480 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac60539-2345-4081-9e21-494be8a9f501 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:05.806787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac60539-2345-4081-9e21-494be8a9f501 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:05.807058 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac60539-2345-4081-9e21-494be8a9f501 None admin] PATH_INFO: `/v3/users/079da78e11b542b5b649433b1528a604` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:05.808135 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac60539-2345-4081-9e21-494be8a9f501 None admin] RBAC: Authorizing `identity:get_user(user_id=079da78e11b542b5b649433b1528a604)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:05.809565 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac60539-2345-4081-9e21-494be8a9f501 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:05.811111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 584/1158] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:29:05 2022] GET /identity/v3/users/079da78e11b542b5b649433b1528a604 => generated 286 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:05.816747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d00e02d-6bb9-4a1c-9bc4-76a6d22eb378 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:05.912646 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d00e02d-6bb9-4a1c-9bc4-76a6d22eb378 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:05.943454 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79393d28-a0a1-4da6-88e6-50c6df15fb8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:05.944033 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79393d28-a0a1-4da6-88e6-50c6df15fb8c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:05.944309 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79393d28-a0a1-4da6-88e6-50c6df15fb8c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:05.945124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79393d28-a0a1-4da6-88e6-50c6df15fb8c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:05.992946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:06.008042 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79393d28-a0a1-4da6-88e6-50c6df15fb8c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:06.010852 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79393d28-a0a1-4da6-88e6-50c6df15fb8c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:06.101487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 575/1159] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:29:05 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 30 09:29:06.107270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac60539-2345-4081-9e21-494be8a9f501 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:06.203619 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac60539-2345-4081-9e21-494be8a9f501 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:06.236091 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b66c8b7-e7c5-42ad-9955-ffe43e26bb03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:06.236708 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b66c8b7-e7c5-42ad-9955-ffe43e26bb03 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:06.236988 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b66c8b7-e7c5-42ad-9955-ffe43e26bb03 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:06.237263 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b66c8b7-e7c5-42ad-9955-ffe43e26bb03 None admin] PATH_INFO: `/v3/projects/7c1dd6274b8a421faa6b4683e26815c3` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:06.238359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b66c8b7-e7c5-42ad-9955-ffe43e26bb03 None admin] RBAC: Authorizing `identity:get_project(project_id=7c1dd6274b8a421faa6b4683e26815c3)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:06.239820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b66c8b7-e7c5-42ad-9955-ffe43e26bb03 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:06.241941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 585/1160] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:29:06 2022] GET /identity/v3/projects/7c1dd6274b8a421faa6b4683e26815c3 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:06.247983 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79393d28-a0a1-4da6-88e6-50c6df15fb8c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:06.328658 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:06.343518 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79393d28-a0a1-4da6-88e6-50c6df15fb8c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:06.376001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3460048f-b300-4ad4-863e-81318893214d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:06.376601 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3460048f-b300-4ad4-863e-81318893214d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:06.376880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3460048f-b300-4ad4-863e-81318893214d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:06.377153 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3460048f-b300-4ad4-863e-81318893214d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:06.377859 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3460048f-b300-4ad4-863e-81318893214d None admin] RBAC: Adding query filter params (role.id=e874c311084c4c9fb7691bce461d319b, scope.project.id=7c1dd6274b8a421faa6b4683e26815c3, user.id=079da78e11b542b5b649433b1528a604) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:29:06.378165 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3460048f-b300-4ad4-863e-81318893214d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:06.379853 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3460048f-b300-4ad4-863e-81318893214d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:06.386020 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 576/1161] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:29:06 2022] GET /identity/v3/role_assignments?user.id=079da78e11b542b5b649433b1528a604&scope.project.id=7c1dd6274b8a421faa6b4683e26815c3&role.id=e874c311084c4c9fb7691bce461d319b => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:08.509734 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2de73c6-0b3e-45e1-b46e-3615abcdf5e0 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:08.510271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2de73c6-0b3e-45e1-b46e-3615abcdf5e0 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:08.510271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2de73c6-0b3e-45e1-b46e-3615abcdf5e0 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:08.511636 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 586/1162] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:29:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:29:08.516958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719a4443-32e8-494d-b1e0-9649cea10b3d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:08.517376 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719a4443-32e8-494d-b1e0-9649cea10b3d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:08.517514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719a4443-32e8-494d-b1e0-9649cea10b3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:08.535655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-719a4443-32e8-494d-b1e0-9649cea10b3d None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:29:08.536831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-719a4443-32e8-494d-b1e0-9649cea10b3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:29:08.599772 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 577/1163] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:29:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:29:08.864671 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2de73c6-0b3e-45e1-b46e-3615abcdf5e0 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:08.908572 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:08.964275 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2de73c6-0b3e-45e1-b46e-3615abcdf5e0 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:08.996427 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ddf9a25-ddac-49f0-96f1-37420af984e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:08.997020 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddf9a25-ddac-49f0-96f1-37420af984e2 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:08.997302 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddf9a25-ddac-49f0-96f1-37420af984e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:08.997576 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddf9a25-ddac-49f0-96f1-37420af984e2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:09.062731 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ddf9a25-ddac-49f0-96f1-37420af984e2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:09.064184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ddf9a25-ddac-49f0-96f1-37420af984e2 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:09.104372 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:09.160104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 587/1164] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:29:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 30 09:29:09.165974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719a4443-32e8-494d-b1e0-9649cea10b3d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:09.240751 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:09.262552 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719a4443-32e8-494d-b1e0-9649cea10b3d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:09.295143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2f65ae-e1a0-4b0f-945a-dc9b83ff6ee3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:09.295768 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2f65ae-e1a0-4b0f-945a-dc9b83ff6ee3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:09.296050 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2f65ae-e1a0-4b0f-945a-dc9b83ff6ee3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:09.296325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2f65ae-e1a0-4b0f-945a-dc9b83ff6ee3 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:09.302752 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2f65ae-e1a0-4b0f-945a-dc9b83ff6ee3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:09.304702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2f65ae-e1a0-4b0f-945a-dc9b83ff6ee3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:09.315907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-1d2f65ae-e1a0-4b0f-945a-dc9b83ff6ee3 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 30 09:29:09.315907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:29:09.315907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 30 09:29:09.315907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 30 09:29:09.315907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:29:09.315907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:29:09.315907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:29:09.315907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:29:09.315907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 30 09:29:09.315907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:29:09.315907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:29:09.315907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:29:09.315907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:29:09.315907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:29:09.315907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:29:09.315907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:29:09.315907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:29:09.317117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:29:09.317117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:29:09.317117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:29:09.317117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:29:09.317117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:29:09.317117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:29:09.317117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:29:09.317117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:29:09.317117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:29:09.317117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:29:09.317117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:29:09.317117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:29:09.317117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:29:09.317117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:29:09.317117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:29:09.317117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:29:09.318279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:29:09.318279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:29:09.318279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:29:09.318279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:29:09.318279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:29:09.318279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:29:09.318279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:29:09.318279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:29:09.318279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:29:09.318279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:29:09.318279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 30 09:29:09.318279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:29:09.318279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 30 09:29:09.318279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:29:09.318279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 578/1165] 10.222.0.39 () {56 vars in 1117 bytes} [Fri Sep 30 09:29:09 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:29:09.323677 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ddf9a25-ddac-49f0-96f1-37420af984e2 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:09.419266 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ddf9a25-ddac-49f0-96f1-37420af984e2 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:09.452824 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44cadba-7e7a-4526-9c89-feda6991b32a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:09.453418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44cadba-7e7a-4526-9c89-feda6991b32a None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:09.453700 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44cadba-7e7a-4526-9c89-feda6991b32a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:09.453976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44cadba-7e7a-4526-9c89-feda6991b32a None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:09.454628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44cadba-7e7a-4526-9c89-feda6991b32a None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:29:09.454939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44cadba-7e7a-4526-9c89-feda6991b32a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:09.456315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44cadba-7e7a-4526-9c89-feda6991b32a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:09.463287 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 588/1166] 10.222.0.39 () {56 vars in 1115 bytes} [Fri Sep 30 09:29:09 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:11.574559 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095d6c5d-1c11-40d7-a918-8c9d8ec270a9 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:11.575155 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095d6c5d-1c11-40d7-a918-8c9d8ec270a9 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:11.575155 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095d6c5d-1c11-40d7-a918-8c9d8ec270a9 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:11.576459 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 579/1167] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:29:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:29:11.581664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09eae320-7f23-4317-95fb-d1328eb07f76 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:11.582064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09eae320-7f23-4317-95fb-d1328eb07f76 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:11.582224 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09eae320-7f23-4317-95fb-d1328eb07f76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:11.595041 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:11.600273 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-09eae320-7f23-4317-95fb-d1328eb07f76 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:29:11.601478 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-09eae320-7f23-4317-95fb-d1328eb07f76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:29:11.666151 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 589/1168] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:29:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:29:11.930260 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095d6c5d-1c11-40d7-a918-8c9d8ec270a9 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:12.027050 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095d6c5d-1c11-40d7-a918-8c9d8ec270a9 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:12.059939 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77240423-5512-445b-8c45-192bad6ed4c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:12.060517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77240423-5512-445b-8c45-192bad6ed4c6 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:12.060793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77240423-5512-445b-8c45-192bad6ed4c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:12.061061 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77240423-5512-445b-8c45-192bad6ed4c6 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:12.061710 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77240423-5512-445b-8c45-192bad6ed4c6 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:12.063090 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77240423-5512-445b-8c45-192bad6ed4c6 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:12.135789 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:12.142363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 580/1169] 10.222.0.39 () {60 vars in 1138 bytes} [Fri Sep 30 09:29:11 2022] POST /identity/v3/domains => generated 267 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:29:14.275062 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cca050-49ef-4c81-9198-098c75cba362 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:14.275538 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cca050-49ef-4c81-9198-098c75cba362 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:14.275648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cca050-49ef-4c81-9198-098c75cba362 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:14.276925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 590/1170] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:29:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:29:14.281977 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa708aa-44b5-4047-be2e-49e847c5124f None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:14.282409 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa708aa-44b5-4047-be2e-49e847c5124f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:14.282512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa708aa-44b5-4047-be2e-49e847c5124f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:14.300100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9aa708aa-44b5-4047-be2e-49e847c5124f None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:29:14.301277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9aa708aa-44b5-4047-be2e-49e847c5124f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:29:14.363983 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 581/1171] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:29:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:29:14.636966 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44cca050-49ef-4c81-9198-098c75cba362 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:14.733586 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44cca050-49ef-4c81-9198-098c75cba362 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:14.766845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f465a4-a528-4768-9715-caad60716da7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:14.767460 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f465a4-a528-4768-9715-caad60716da7 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:14.767739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f465a4-a528-4768-9715-caad60716da7 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:14.768010 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f465a4-a528-4768-9715-caad60716da7 None admin] PATH_INFO: `/v3/domains/99c7cb5c54fe4a0098a8bb231b02f56d` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:14.770557 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:14.778291 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f465a4-a528-4768-9715-caad60716da7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=99c7cb5c54fe4a0098a8bb231b02f56d)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:14.779768 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f465a4-a528-4768-9715-caad60716da7 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:14.781319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 591/1172] 10.222.0.39 () {56 vars in 1183 bytes} [Fri Sep 30 09:29:14 2022] GET /identity/v3/domains/99c7cb5c54fe4a0098a8bb231b02f56d => generated 267 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:14.786916 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa708aa-44b5-4047-be2e-49e847c5124f None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:14.883158 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa708aa-44b5-4047-be2e-49e847c5124f None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:14.907959 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:14.915888 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d4815f-538e-45c3-bba3-95b33f0ad8e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:14.916897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d4815f-538e-45c3-bba3-95b33f0ad8e5 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:14.916897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d4815f-538e-45c3-bba3-95b33f0ad8e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:14.917113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d4815f-538e-45c3-bba3-95b33f0ad8e5 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:14.917761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d4815f-538e-45c3-bba3-95b33f0ad8e5 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:14.919210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d4815f-538e-45c3-bba3-95b33f0ad8e5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:15.002266 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 582/1173] 10.222.0.39 () {60 vars in 1141 bytes} [Fri Sep 30 09:29:14 2022] POST /identity/v3/projects => generated 361 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:29:17.172225 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7bbf07-936d-4b71-b57c-6c78ee1a65eb None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:17.172753 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7bbf07-936d-4b71-b57c-6c78ee1a65eb None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:17.172753 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7bbf07-936d-4b71-b57c-6c78ee1a65eb None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:17.174102 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 592/1174] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:29:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:29:17.179382 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd8bd80-5134-4e92-8a0d-c44a390e08d3 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:17.179803 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd8bd80-5134-4e92-8a0d-c44a390e08d3 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:17.179995 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd8bd80-5134-4e92-8a0d-c44a390e08d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:17.198481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-fdd8bd80-5134-4e92-8a0d-c44a390e08d3 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:29:17.199289 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fdd8bd80-5134-4e92-8a0d-c44a390e08d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:29:17.262742 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 583/1175] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:29:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:29:17.532737 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7bbf07-936d-4b71-b57c-6c78ee1a65eb None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:17.631009 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7bbf07-936d-4b71-b57c-6c78ee1a65eb None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:17.663448 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4b5e5e-dc9b-4ad0-a7b2-ac89a63dbe66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:17.664058 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4b5e5e-dc9b-4ad0-a7b2-ac89a63dbe66 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:17.664337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4b5e5e-dc9b-4ad0-a7b2-ac89a63dbe66 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:17.664607 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4b5e5e-dc9b-4ad0-a7b2-ac89a63dbe66 None admin] PATH_INFO: `/v3/domains/99c7cb5c54fe4a0098a8bb231b02f56d` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:17.665701 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4b5e5e-dc9b-4ad0-a7b2-ac89a63dbe66 None admin] RBAC: Authorizing `identity:get_domain(domain_id=99c7cb5c54fe4a0098a8bb231b02f56d)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:17.667167 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4b5e5e-dc9b-4ad0-a7b2-ac89a63dbe66 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:17.668693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 593/1176] 10.222.0.39 () {56 vars in 1183 bytes} [Fri Sep 30 09:29:17 2022] GET /identity/v3/domains/99c7cb5c54fe4a0098a8bb231b02f56d => generated 267 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:17.674204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd8bd80-5134-4e92-8a0d-c44a390e08d3 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:17.768708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd8bd80-5134-4e92-8a0d-c44a390e08d3 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:17.778750 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:17.801422 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5bbd0a-72e8-4dfa-b0ff-61c7d820ac10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:17.801968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5bbd0a-72e8-4dfa-b0ff-61c7d820ac10 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:17.802252 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5bbd0a-72e8-4dfa-b0ff-61c7d820ac10 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:17.802663 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5bbd0a-72e8-4dfa-b0ff-61c7d820ac10 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:17.803852 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5bbd0a-72e8-4dfa-b0ff-61c7d820ac10 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:17.805278 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5bbd0a-72e8-4dfa-b0ff-61c7d820ac10 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:17.906224 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 584/1177] 10.222.0.39 () {60 vars in 1132 bytes} [Fri Sep 30 09:29:17 2022] POST /identity/v3/users => generated 311 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:29:20.093448 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09811951-f220-41e5-84ff-d0aca38d6989 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:20.094006 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09811951-f220-41e5-84ff-d0aca38d6989 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:20.094006 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09811951-f220-41e5-84ff-d0aca38d6989 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:20.095358 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 594/1178] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:29:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:29:20.100980 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16fe875-bae8-4c09-96ad-85af1e3d2c2a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:20.101411 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16fe875-bae8-4c09-96ad-85af1e3d2c2a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:20.101518 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16fe875-bae8-4c09-96ad-85af1e3d2c2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:20.119865 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e16fe875-bae8-4c09-96ad-85af1e3d2c2a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:29:20.121083 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e16fe875-bae8-4c09-96ad-85af1e3d2c2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:29:20.186650 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 585/1179] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:29:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:29:20.457561 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09811951-f220-41e5-84ff-d0aca38d6989 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:20.519472 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:20.557248 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09811951-f220-41e5-84ff-d0aca38d6989 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:20.590356 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44653f16-8192-4921-8e13-5eb919b6fbac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:20.590997 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44653f16-8192-4921-8e13-5eb919b6fbac None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:20.591277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44653f16-8192-4921-8e13-5eb919b6fbac None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:20.591548 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44653f16-8192-4921-8e13-5eb919b6fbac None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:20.601351 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44653f16-8192-4921-8e13-5eb919b6fbac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:29:20.606118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-44653f16-8192-4921-8e13-5eb919b6fbac None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:29:20.606118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:29:20.606118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:29:20.606118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:29:20.606118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:29:20.606118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:29:20.606118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:29:20.606118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:29:20.606118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:29:20.606118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:29:20.606118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:29:20.606118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:29:20.606118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:29:20.606118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:29:20.606118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:29:20.606118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:29:20.607423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:29:20.607423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:29:20.607423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:29:20.607423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:29:20.607423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:29:20.607423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:29:20.607423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:29:20.607423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:29:20.607423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:29:20.607423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:29:20.607423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:29:20.607423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:29:20.607423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:29:20.607423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:29:20.607423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:29:20.607423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:29:20.608606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:29:20.608606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:29:20.608606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:29:20.608606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:29:20.608606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:29:20.608606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:29:20.608606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:29:20.608606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:29:20.608606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:29:20.608606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:29:20.608606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:29:20.608606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:29:20.608606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 595/1180] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:29:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:29:20.613272 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16fe875-bae8-4c09-96ad-85af1e3d2c2a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:20.703154 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:20.713397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16fe875-bae8-4c09-96ad-85af1e3d2c2a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:20.741329 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb122be5-618f-4864-b47a-1762841a6590 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:20.741923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb122be5-618f-4864-b47a-1762841a6590 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:20.742209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb122be5-618f-4864-b47a-1762841a6590 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:20.742479 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb122be5-618f-4864-b47a-1762841a6590 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:20.743160 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb122be5-618f-4864-b47a-1762841a6590 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:29:20.743464 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb122be5-618f-4864-b47a-1762841a6590 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:20.744814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb122be5-618f-4864-b47a-1762841a6590 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:20.752432 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 586/1181] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:29:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:20.759085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44653f16-8192-4921-8e13-5eb919b6fbac None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:20.856573 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44653f16-8192-4921-8e13-5eb919b6fbac None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:20.879645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:20.889616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243d836e-00f4-4600-bb92-bece73a73570 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:20.890762 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243d836e-00f4-4600-bb92-bece73a73570 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:20.890762 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243d836e-00f4-4600-bb92-bece73a73570 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:20.890762 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243d836e-00f4-4600-bb92-bece73a73570 None admin] PATH_INFO: `/v3/users/5e0cda64683d4fd7b13ba11c47de3b99` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:20.903092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243d836e-00f4-4600-bb92-bece73a73570 None admin] RBAC: Authorizing `identity:get_user(user_id=5e0cda64683d4fd7b13ba11c47de3b99)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:20.904535 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243d836e-00f4-4600-bb92-bece73a73570 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:20.906081 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 596/1182] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:29:20 2022] GET /identity/v3/users/5e0cda64683d4fd7b13ba11c47de3b99 => generated 311 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:20.911820 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb122be5-618f-4864-b47a-1762841a6590 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:21.007233 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb122be5-618f-4864-b47a-1762841a6590 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:21.038824 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a65ed59-713e-47cd-b9a3-5ed08d9ecd7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:21.039410 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a65ed59-713e-47cd-b9a3-5ed08d9ecd7b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:21.039691 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a65ed59-713e-47cd-b9a3-5ed08d9ecd7b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:21.039964 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a65ed59-713e-47cd-b9a3-5ed08d9ecd7b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:21.065312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:21.103780 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a65ed59-713e-47cd-b9a3-5ed08d9ecd7b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:21.105206 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a65ed59-713e-47cd-b9a3-5ed08d9ecd7b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:21.197980 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 587/1183] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:29:20 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 287 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 30 09:29:21.203815 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243d836e-00f4-4600-bb92-bece73a73570 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:21.298900 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243d836e-00f4-4600-bb92-bece73a73570 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:21.330954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29229c4-d130-4c96-85d8-b36fc82a5c0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:21.331563 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29229c4-d130-4c96-85d8-b36fc82a5c0b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:21.331864 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29229c4-d130-4c96-85d8-b36fc82a5c0b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:21.332138 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29229c4-d130-4c96-85d8-b36fc82a5c0b None admin] PATH_INFO: `/v3/projects/a4b000ee237e403f9b9ce4970dd76fd4` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:21.333218 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29229c4-d130-4c96-85d8-b36fc82a5c0b None admin] RBAC: Authorizing `identity:get_project(project_id=a4b000ee237e403f9b9ce4970dd76fd4)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:21.335261 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29229c4-d130-4c96-85d8-b36fc82a5c0b None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:21.336831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 597/1184] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:29:21 2022] GET /identity/v3/projects/a4b000ee237e403f9b9ce4970dd76fd4 => generated 361 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:21.342838 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a65ed59-713e-47cd-b9a3-5ed08d9ecd7b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:21.397911 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:21.436949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a65ed59-713e-47cd-b9a3-5ed08d9ecd7b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:21.469726 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacb655d-61e7-4ac6-812b-ea408b843464 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:21.470305 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacb655d-61e7-4ac6-812b-ea408b843464 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:21.470580 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacb655d-61e7-4ac6-812b-ea408b843464 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:21.470938 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacb655d-61e7-4ac6-812b-ea408b843464 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:21.471639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacb655d-61e7-4ac6-812b-ea408b843464 None admin] RBAC: Adding query filter params (role.id=e874c311084c4c9fb7691bce461d319b, scope.project.id=a4b000ee237e403f9b9ce4970dd76fd4, user.id=5e0cda64683d4fd7b13ba11c47de3b99) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:29:21.471941 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacb655d-61e7-4ac6-812b-ea408b843464 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:21.473339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacb655d-61e7-4ac6-812b-ea408b843464 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:21.479193 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 588/1185] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:29:21 2022] GET /identity/v3/role_assignments?user.id=5e0cda64683d4fd7b13ba11c47de3b99&scope.project.id=a4b000ee237e403f9b9ce4970dd76fd4&role.id=e874c311084c4c9fb7691bce461d319b => generated 259 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:23.561005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b39b88-1808-455f-886c-78c85f4440aa None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:23.561527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b39b88-1808-455f-886c-78c85f4440aa None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:23.561527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b39b88-1808-455f-886c-78c85f4440aa None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:23.562909 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 598/1186] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:29:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:29:23.567924 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142aade0-1633-48f5-8ed5-49952bada836 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:23.568925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142aade0-1633-48f5-8ed5-49952bada836 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:23.569016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142aade0-1633-48f5-8ed5-49952bada836 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:23.586751 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-142aade0-1633-48f5-8ed5-49952bada836 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:29:23.587943 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-142aade0-1633-48f5-8ed5-49952bada836 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:29:23.651890 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 589/1187] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:29:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:29:23.914385 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b39b88-1808-455f-886c-78c85f4440aa None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:23.964914 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:24.013514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b39b88-1808-455f-886c-78c85f4440aa None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:24.045303 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd71770-a8bc-4169-9a0d-6639e0f49819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:24.045910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd71770-a8bc-4169-9a0d-6639e0f49819 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:24.046185 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd71770-a8bc-4169-9a0d-6639e0f49819 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:24.046454 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd71770-a8bc-4169-9a0d-6639e0f49819 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:24.057150 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd71770-a8bc-4169-9a0d-6639e0f49819 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:29:24.061860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-efd71770-a8bc-4169-9a0d-6639e0f49819 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:29:24.061860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:29:24.061860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:29:24.061860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:29:24.061860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:29:24.061860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:29:24.061860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:29:24.061860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:29:24.061860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:29:24.061860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:29:24.061860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:29:24.061860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:29:24.061860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:29:24.061860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:29:24.061860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:29:24.061860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:29:24.062967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:29:24.062967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:29:24.062967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:29:24.062967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:29:24.062967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:29:24.062967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:29:24.062967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:29:24.062967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:29:24.062967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:29:24.062967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:29:24.062967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:29:24.062967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:29:24.062967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:29:24.062967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:29:24.062967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:29:24.062967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:29:24.064015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:29:24.064015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:29:24.064015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:29:24.064015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:29:24.064015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:29:24.064015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:29:24.064015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:29:24.064015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:29:24.064015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:29:24.064015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:29:24.064015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:29:24.064015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:29:24.064015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 599/1188] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:29:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:29:24.069018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142aade0-1633-48f5-8ed5-49952bada836 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:24.119684 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:24.165024 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142aade0-1633-48f5-8ed5-49952bada836 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:24.196419 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f233ead3-4e6f-4627-a59f-baa49bca5319 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:24.197025 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f233ead3-4e6f-4627-a59f-baa49bca5319 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:24.197302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f233ead3-4e6f-4627-a59f-baa49bca5319 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:24.197919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f233ead3-4e6f-4627-a59f-baa49bca5319 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:24.198221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f233ead3-4e6f-4627-a59f-baa49bca5319 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:29:24.198523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f233ead3-4e6f-4627-a59f-baa49bca5319 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:24.199900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f233ead3-4e6f-4627-a59f-baa49bca5319 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:24.208084 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 590/1189] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:29:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:24.214313 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd71770-a8bc-4169-9a0d-6639e0f49819 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:24.310124 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd71770-a8bc-4169-9a0d-6639e0f49819 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:24.319836 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:24.342168 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6581f65f-cff9-461e-9326-2b4821f56416 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:24.342813 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6581f65f-cff9-461e-9326-2b4821f56416 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:24.343133 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6581f65f-cff9-461e-9326-2b4821f56416 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:24.343422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6581f65f-cff9-461e-9326-2b4821f56416 None admin] PATH_INFO: `/v3/users/5e0cda64683d4fd7b13ba11c47de3b99` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:24.344511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6581f65f-cff9-461e-9326-2b4821f56416 None admin] RBAC: Authorizing `identity:get_user(user_id=5e0cda64683d4fd7b13ba11c47de3b99)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:24.345928 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6581f65f-cff9-461e-9326-2b4821f56416 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:24.347474 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 600/1190] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:29:24 2022] GET /identity/v3/users/5e0cda64683d4fd7b13ba11c47de3b99 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:24.353755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f233ead3-4e6f-4627-a59f-baa49bca5319 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:24.447894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f233ead3-4e6f-4627-a59f-baa49bca5319 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:24.461479 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:24.480578 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b074e1f5-bc8d-4bef-8ec0-94c2ba029109 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:24.481180 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b074e1f5-bc8d-4bef-8ec0-94c2ba029109 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:24.481460 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b074e1f5-bc8d-4bef-8ec0-94c2ba029109 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:24.481731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b074e1f5-bc8d-4bef-8ec0-94c2ba029109 None admin] PATH_INFO: `/v3/projects/a4b000ee237e403f9b9ce4970dd76fd4` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:24.482853 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b074e1f5-bc8d-4bef-8ec0-94c2ba029109 None admin] RBAC: Authorizing `identity:get_project(project_id=a4b000ee237e403f9b9ce4970dd76fd4)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:24.484274 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b074e1f5-bc8d-4bef-8ec0-94c2ba029109 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:24.485837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 591/1191] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:29:24 2022] GET /identity/v3/projects/a4b000ee237e403f9b9ce4970dd76fd4 => generated 361 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:24.491562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6581f65f-cff9-461e-9326-2b4821f56416 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:24.587194 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6581f65f-cff9-461e-9326-2b4821f56416 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:24.618841 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5949e243-2274-4e0e-bdde-3a2538ff339a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:24.619479 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5949e243-2274-4e0e-bdde-3a2538ff339a None admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:24.619755 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5949e243-2274-4e0e-bdde-3a2538ff339a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:24.620021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5949e243-2274-4e0e-bdde-3a2538ff339a None admin] PATH_INFO: `/v3/projects/a4b000ee237e403f9b9ce4970dd76fd4/users/5e0cda64683d4fd7b13ba11c47de3b99/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:24.621862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5949e243-2274-4e0e-bdde-3a2538ff339a None admin] RBAC: Authorizing `identity:create_grant(project_id=a4b000ee237e403f9b9ce4970dd76fd4, user_id=5e0cda64683d4fd7b13ba11c47de3b99, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:24.623317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5949e243-2274-4e0e-bdde-3a2538ff339a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:24.698048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 601/1192] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:29:24 2022] PUT /identity/v3/projects/a4b000ee237e403f9b9ce4970dd76fd4/users/5e0cda64683d4fd7b13ba11c47de3b99/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:29:26.780059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0a2f25-42d1-4242-bbba-7b1a1b6b1403 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:26.780625 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0a2f25-42d1-4242-bbba-7b1a1b6b1403 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:26.780625 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0a2f25-42d1-4242-bbba-7b1a1b6b1403 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:26.781943 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 592/1193] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:29:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:29:26.787815 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2c6ae6-7abc-4034-85a0-db5119a208be None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:26.788221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2c6ae6-7abc-4034-85a0-db5119a208be None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:26.788370 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2c6ae6-7abc-4034-85a0-db5119a208be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:26.806666 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-eb2c6ae6-7abc-4034-85a0-db5119a208be None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:29:26.807856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-eb2c6ae6-7abc-4034-85a0-db5119a208be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:29:26.835795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:26.871515 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 602/1194] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:29:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:29:27.137784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0a2f25-42d1-4242-bbba-7b1a1b6b1403 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:27.236535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0a2f25-42d1-4242-bbba-7b1a1b6b1403 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:27.269603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57611ae6-1a83-43e4-ad30-50c88cd97ea7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:27.270224 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57611ae6-1a83-43e4-ad30-50c88cd97ea7 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:27.270506 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57611ae6-1a83-43e4-ad30-50c88cd97ea7 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:27.271291 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57611ae6-1a83-43e4-ad30-50c88cd97ea7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:27.281080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57611ae6-1a83-43e4-ad30-50c88cd97ea7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:29:27.285880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-57611ae6-1a83-43e4-ad30-50c88cd97ea7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:29:27.285880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:29:27.285880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:29:27.285880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:29:27.285880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:29:27.285880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:29:27.285880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:29:27.285880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:29:27.285880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:29:27.285880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:29:27.285880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:29:27.285880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:29:27.285880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:29:27.285880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:29:27.285880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:29:27.285880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:29:27.287543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:29:27.287543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:29:27.287543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:29:27.287543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:29:27.287543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:29:27.287543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:29:27.287543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:29:27.287543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:29:27.287543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:29:27.287543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:29:27.287543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:29:27.287543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:29:27.287543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:29:27.287543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:29:27.287543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:29:27.287543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:29:27.288695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:29:27.288695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:29:27.288695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:29:27.288695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:29:27.288695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:29:27.288695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:29:27.288695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:29:27.288695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:29:27.288695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:29:27.288695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:29:27.288695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:29:27.288695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:29:27.288695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 593/1195] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:29:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:29:27.293478 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2c6ae6-7abc-4034-85a0-db5119a208be None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:27.386992 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2c6ae6-7abc-4034-85a0-db5119a208be None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:27.419510 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403ec3f3-8394-45a2-bfc3-3ebdaa6af621 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:27.420042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403ec3f3-8394-45a2-bfc3-3ebdaa6af621 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:27.420317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403ec3f3-8394-45a2-bfc3-3ebdaa6af621 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:27.420584 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403ec3f3-8394-45a2-bfc3-3ebdaa6af621 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:27.421233 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403ec3f3-8394-45a2-bfc3-3ebdaa6af621 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:29:27.421526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403ec3f3-8394-45a2-bfc3-3ebdaa6af621 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:27.422904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403ec3f3-8394-45a2-bfc3-3ebdaa6af621 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:27.430503 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 603/1196] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:29:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:27.436391 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57611ae6-1a83-43e4-ad30-50c88cd97ea7 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:27.479503 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:27.533866 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57611ae6-1a83-43e4-ad30-50c88cd97ea7 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:27.566542 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52270f2b-fa8d-422b-b93f-21b2006713a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:27.567189 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52270f2b-fa8d-422b-b93f-21b2006713a3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:27.567473 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52270f2b-fa8d-422b-b93f-21b2006713a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:27.568277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52270f2b-fa8d-422b-b93f-21b2006713a3 None admin] PATH_INFO: `/v3/users/5e0cda64683d4fd7b13ba11c47de3b99` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:27.569389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52270f2b-fa8d-422b-b93f-21b2006713a3 None admin] RBAC: Authorizing `identity:get_user(user_id=5e0cda64683d4fd7b13ba11c47de3b99)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:27.570862 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52270f2b-fa8d-422b-b93f-21b2006713a3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:27.572420 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 594/1197] 10.222.0.39 () {56 vars in 1177 bytes} [Fri Sep 30 09:29:27 2022] GET /identity/v3/users/5e0cda64683d4fd7b13ba11c47de3b99 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:27.578216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403ec3f3-8394-45a2-bfc3-3ebdaa6af621 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:27.608341 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:27.677029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403ec3f3-8394-45a2-bfc3-3ebdaa6af621 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:27.709978 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d2a81c-9f44-44c1-bf2d-85573edfcb1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:27.710568 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d2a81c-9f44-44c1-bf2d-85573edfcb1e None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:27.710874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d2a81c-9f44-44c1-bf2d-85573edfcb1e None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:27.711142 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d2a81c-9f44-44c1-bf2d-85573edfcb1e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:27.775271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d2a81c-9f44-44c1-bf2d-85573edfcb1e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:27.776719 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d2a81c-9f44-44c1-bf2d-85573edfcb1e None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:27.803764 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:27.870898 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 604/1198] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:29:27 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 30 09:29:27.876892 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52270f2b-fa8d-422b-b93f-21b2006713a3 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:27.972694 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52270f2b-fa8d-422b-b93f-21b2006713a3 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:27.982419 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:28.005741 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86873d2b-e840-4c96-a38f-19bff2706217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:28.006369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86873d2b-e840-4c96-a38f-19bff2706217 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:28.006677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86873d2b-e840-4c96-a38f-19bff2706217 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:28.006956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86873d2b-e840-4c96-a38f-19bff2706217 None admin] PATH_INFO: `/v3/projects/a4b000ee237e403f9b9ce4970dd76fd4` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:28.008057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86873d2b-e840-4c96-a38f-19bff2706217 None admin] RBAC: Authorizing `identity:get_project(project_id=a4b000ee237e403f9b9ce4970dd76fd4)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:28.009472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86873d2b-e840-4c96-a38f-19bff2706217 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:28.011103 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 595/1199] 10.222.0.39 () {56 vars in 1186 bytes} [Fri Sep 30 09:29:27 2022] GET /identity/v3/projects/a4b000ee237e403f9b9ce4970dd76fd4 => generated 361 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:28.017120 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d2a81c-9f44-44c1-bf2d-85573edfcb1e None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:29:28.110688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d2a81c-9f44-44c1-bf2d-85573edfcb1e None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:29:28.138406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:28.143694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67c7438-ac86-409a-8fae-d7b71659b392 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:29:28.144276 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67c7438-ac86-409a-8fae-d7b71659b392 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:29:28.144549 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67c7438-ac86-409a-8fae-d7b71659b392 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:29:28.144813 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67c7438-ac86-409a-8fae-d7b71659b392 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:29:28.145514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67c7438-ac86-409a-8fae-d7b71659b392 None admin] RBAC: Adding query filter params (role.id=e874c311084c4c9fb7691bce461d319b, scope.project.id=a4b000ee237e403f9b9ce4970dd76fd4, user.id=5e0cda64683d4fd7b13ba11c47de3b99) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:29:28.145883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67c7438-ac86-409a-8fae-d7b71659b392 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:29:28.147465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67c7438-ac86-409a-8fae-d7b71659b392 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:29:28.153001 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 605/1200] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:29:28 2022] GET /identity/v3/role_assignments?user.id=5e0cda64683d4fd7b13ba11c47de3b99&scope.project.id=a4b000ee237e403f9b9ce4970dd76fd4&role.id=e874c311084c4c9fb7691bce461d319b => generated 608 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:29:38.014257 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=11 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:29:38.163118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:40.079314 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1530c49d-2809-40e1-ae98-121e1e5f9aa7 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:40.079910 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1530c49d-2809-40e1-ae98-121e1e5f9aa7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:40.079910 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1530c49d-2809-40e1-ae98-121e1e5f9aa7 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:40.081225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 596/1201] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:30:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:30:40.087071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b74152e-e12a-4bd6-87c0-5a8bda2e49ba None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:40.087510 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b74152e-e12a-4bd6-87c0-5a8bda2e49ba None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:40.087694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b74152e-e12a-4bd6-87c0-5a8bda2e49ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:40.105915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-5b74152e-e12a-4bd6-87c0-5a8bda2e49ba None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:30:40.107117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-5b74152e-e12a-4bd6-87c0-5a8bda2e49ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:30:40.171960 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 606/1202] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:30:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 30 09:30:40.447200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1530c49d-2809-40e1-ae98-121e1e5f9aa7 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:40.546776 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1530c49d-2809-40e1-ae98-121e1e5f9aa7 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:40.579731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697d1a83-c36d-4374-9f37-8942c4b9a744 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:40.580365 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697d1a83-c36d-4374-9f37-8942c4b9a744 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:40.580642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697d1a83-c36d-4374-9f37-8942c4b9a744 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:40.580911 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697d1a83-c36d-4374-9f37-8942c4b9a744 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:40.590573 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697d1a83-c36d-4374-9f37-8942c4b9a744 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:40.595995 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697d1a83-c36d-4374-9f37-8942c4b9a744 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:40.606151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-697d1a83-c36d-4374-9f37-8942c4b9a744 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:30:40.606151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:40.606151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:40.606151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:40.606151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:40.606151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:40.606151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:40.606151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:40.606151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:40.606151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:40.606151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:40.606151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:40.606151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:30:40.606151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:30:40.606151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:30:40.606151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:30:40.607255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:40.607255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:40.607255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:40.607255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:40.607255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:40.607255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:40.607255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:40.607255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:40.607255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:40.607255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:40.607255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:40.607255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:40.607255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:40.607255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:40.607255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:40.607255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:40.607255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:30:40.608414 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:30:40.608414 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:30:40.608414 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:30:40.608414 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 597/1203] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:30:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:30:40.613618 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b74152e-e12a-4bd6-87c0-5a8bda2e49ba None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:40.710581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b74152e-e12a-4bd6-87c0-5a8bda2e49ba None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:40.737998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:40.745525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afc5a15-574c-4f42-8540-769720a58b01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:40.746221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afc5a15-574c-4f42-8540-769720a58b01 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:40.746417 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afc5a15-574c-4f42-8540-769720a58b01 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:40.746714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afc5a15-574c-4f42-8540-769720a58b01 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:40.747381 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afc5a15-574c-4f42-8540-769720a58b01 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:40.747681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afc5a15-574c-4f42-8540-769720a58b01 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:40.749048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afc5a15-574c-4f42-8540-769720a58b01 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:40.760979 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 607/1204] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:30:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:30:40.766833 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697d1a83-c36d-4374-9f37-8942c4b9a744 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:40.818865 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:40.863113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697d1a83-c36d-4374-9f37-8942c4b9a744 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:40.895892 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c2f797-23fc-4059-9c98-114fc26990e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:40.896489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c2f797-23fc-4059-9c98-114fc26990e9 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:40.896763 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c2f797-23fc-4059-9c98-114fc26990e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:40.897032 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c2f797-23fc-4059-9c98-114fc26990e9 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:40.897661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c2f797-23fc-4059-9c98-114fc26990e9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:40.899150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c2f797-23fc-4059-9c98-114fc26990e9 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:40.998401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 598/1205] 10.222.0.39 () {60 vars in 1132 bytes} [Fri Sep 30 09:30:40 2022] POST /identity/v3/users => generated 251 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:30:43.142163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2266f2-d2c5-4bf7-8d9d-a576c4770286 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:43.142873 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2266f2-d2c5-4bf7-8d9d-a576c4770286 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:43.142873 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2266f2-d2c5-4bf7-8d9d-a576c4770286 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:43.144147 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 608/1206] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:30:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:30:43.149397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d999753-ae26-4b30-901f-27b408e1149a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:43.149817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d999753-ae26-4b30-901f-27b408e1149a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:43.149954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d999753-ae26-4b30-901f-27b408e1149a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:43.168263 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-1d999753-ae26-4b30-901f-27b408e1149a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:30:43.169441 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1d999753-ae26-4b30-901f-27b408e1149a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:30:43.234366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 599/1207] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:30:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:30:43.500805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2266f2-d2c5-4bf7-8d9d-a576c4770286 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:43.601659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2266f2-d2c5-4bf7-8d9d-a576c4770286 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:43.634407 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3dc09d-2019-4db1-a9bd-bc3c852d667f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:43.635052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3dc09d-2019-4db1-a9bd-bc3c852d667f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:43.635333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3dc09d-2019-4db1-a9bd-bc3c852d667f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:43.635604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3dc09d-2019-4db1-a9bd-bc3c852d667f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:43.645369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3dc09d-2019-4db1-a9bd-bc3c852d667f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:30:43.650681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-df3dc09d-2019-4db1-a9bd-bc3c852d667f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:30:43.650681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:43.650681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:43.650681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:43.650681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:43.650681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:43.650681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:43.650681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:43.650681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:43.650681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:43.650681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:43.650681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:43.650681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:30:43.650681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:30:43.650681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:30:43.650681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:30:43.653318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:30:43.653318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:30:43.653318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:30:43.653318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:30:43.653318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:43.653318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:43.653318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:43.653318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:43.653318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:43.653318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:43.653318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:43.653318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:43.653318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:43.653318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:43.653318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:43.653318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:43.654352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:43.654352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:43.654352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:43.654352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:43.654352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:30:43.654352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:30:43.654352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:30:43.654352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:30:43.654352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:30:43.654352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:30:43.654352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:30:43.654352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:30:43.654352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 609/1208] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:30:43 2022] GET /identity/v3/roles/service => generated 85 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:30:43.657756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d999753-ae26-4b30-901f-27b408e1149a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:43.695539 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:43.753225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d999753-ae26-4b30-901f-27b408e1149a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:43.785922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ede0d1-e920-4f9b-9b69-38a8abaa3fd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:43.787179 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ede0d1-e920-4f9b-9b69-38a8abaa3fd3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:43.787454 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ede0d1-e920-4f9b-9b69-38a8abaa3fd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:43.787720 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ede0d1-e920-4f9b-9b69-38a8abaa3fd3 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:43.788361 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ede0d1-e920-4f9b-9b69-38a8abaa3fd3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:43.788657 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ede0d1-e920-4f9b-9b69-38a8abaa3fd3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:43.790033 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ede0d1-e920-4f9b-9b69-38a8abaa3fd3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:43.798558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 600/1209] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:30:43 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:43.804734 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3dc09d-2019-4db1-a9bd-bc3c852d667f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:43.856479 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:43.907286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3dc09d-2019-4db1-a9bd-bc3c852d667f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:43.940485 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0636be45-3026-4647-aa60-c9b3f4fa383c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:43.941105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0636be45-3026-4647-aa60-c9b3f4fa383c None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:43.941381 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0636be45-3026-4647-aa60-c9b3f4fa383c None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:43.941646 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0636be45-3026-4647-aa60-c9b3f4fa383c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:43.952341 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0636be45-3026-4647-aa60-c9b3f4fa383c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:43.953777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0636be45-3026-4647-aa60-c9b3f4fa383c None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:43.968139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-0636be45-3026-4647-aa60-c9b3f4fa383c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:30:43.968139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:43.968139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:43.968139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:43.968139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:43.968139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:43.968139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:43.968139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:43.968139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:43.968139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:43.968139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:43.968139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:43.968139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:30:43.968139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:30:43.968139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:30:43.968139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:30:43.969297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:43.969297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:43.969297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:43.969297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:43.969297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:43.969297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:43.969297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:43.969297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:43.969297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:43.969297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:43.969297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:43.969297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:43.969297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:43.969297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:43.969297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:43.969297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:43.969297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:30:43.970247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:30:43.970247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:30:43.970247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:30:43.970247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 610/1210] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:30:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:30:43.975271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ede0d1-e920-4f9b-9b69-38a8abaa3fd3 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:44.069398 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:44.072167 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ede0d1-e920-4f9b-9b69-38a8abaa3fd3 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:44.105148 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b21c310-f483-474f-99e8-5fc98790f6ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:44.105740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b21c310-f483-474f-99e8-5fc98790f6ad None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:44.106013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b21c310-f483-474f-99e8-5fc98790f6ad None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:44.106279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b21c310-f483-474f-99e8-5fc98790f6ad None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:44.106928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b21c310-f483-474f-99e8-5fc98790f6ad None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:44.107226 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b21c310-f483-474f-99e8-5fc98790f6ad None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:44.108572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b21c310-f483-474f-99e8-5fc98790f6ad None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:44.119321 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 601/1211] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:30:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:44.125476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0636be45-3026-4647-aa60-c9b3f4fa383c None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:44.224758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0636be45-3026-4647-aa60-c9b3f4fa383c None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:44.226250 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:44.258596 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7d5bc4-fd96-4496-b0b7-2c88a9b8098f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:44.259233 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7d5bc4-fd96-4496-b0b7-2c88a9b8098f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:44.259509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7d5bc4-fd96-4496-b0b7-2c88a9b8098f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:44.259815 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7d5bc4-fd96-4496-b0b7-2c88a9b8098f None admin] PATH_INFO: `/v3/users/placement` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:44.266412 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7d5bc4-fd96-4496-b0b7-2c88a9b8098f None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:44.267882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7d5bc4-fd96-4496-b0b7-2c88a9b8098f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:44.279155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-6c7d5bc4-fd96-4496-b0b7-2c88a9b8098f None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 30 09:30:44.279155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:44.279155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:44.279155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:44.279155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:44.279155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:44.279155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:44.279155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:44.279155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:44.279155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:44.279155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:44.279155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:44.279155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:30:44.279155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:30:44.279155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:30:44.279155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:30:44.280389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:44.280389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:44.280389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:30:44.280389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:30:44.280389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:30:44.280389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:30:44.280389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:44.280389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:44.280389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:44.280389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:44.280389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:44.280389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:44.280389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:44.280389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:44.280389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:44.280389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:44.280389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:44.280389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:44.281553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:44.281553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:44.281553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:30:44.281553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:30:44.281553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:30:44.281553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:30:44.281553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:30:44.281553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:30:44.281553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 30 09:30:44.281553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:30:44.281553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 611/1212] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:30:44 2022] GET /identity/v3/users/placement => generated 87 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:30:44.286078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b21c310-f483-474f-99e8-5fc98790f6ad None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:44.385119 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b21c310-f483-474f-99e8-5fc98790f6ad None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:44.417078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4528e9-1ad3-42b4-bce7-15c076e95d9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:44.417695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4528e9-1ad3-42b4-bce7-15c076e95d9b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:44.417973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4528e9-1ad3-42b4-bce7-15c076e95d9b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:44.418248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4528e9-1ad3-42b4-bce7-15c076e95d9b None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:44.418984 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4528e9-1ad3-42b4-bce7-15c076e95d9b None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:44.419336 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4528e9-1ad3-42b4-bce7-15c076e95d9b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:44.420711 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4528e9-1ad3-42b4-bce7-15c076e95d9b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:44.437841 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 602/1213] 10.222.0.39 () {56 vars in 1143 bytes} [Fri Sep 30 09:30:44 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:44.444394 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7d5bc4-fd96-4496-b0b7-2c88a9b8098f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:44.543570 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7d5bc4-fd96-4496-b0b7-2c88a9b8098f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:44.576065 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e2c4d1-b8c4-4f1f-b3dd-b24a5fb64426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:44.576656 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e2c4d1-b8c4-4f1f-b3dd-b24a5fb64426 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:44.576933 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e2c4d1-b8c4-4f1f-b3dd-b24a5fb64426 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:44.577202 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e2c4d1-b8c4-4f1f-b3dd-b24a5fb64426 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:44.605525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:44.642255 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e2c4d1-b8c4-4f1f-b3dd-b24a5fb64426 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:44.643655 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e2c4d1-b8c4-4f1f-b3dd-b24a5fb64426 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:44.738265 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 612/1214] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:30:44 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 30 09:30:44.744607 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4528e9-1ad3-42b4-bce7-15c076e95d9b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:44.763596 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:44.841276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4528e9-1ad3-42b4-bce7-15c076e95d9b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:44.875174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a308af94-370a-46d7-bc30-1aeab5dbba3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:44.875792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a308af94-370a-46d7-bc30-1aeab5dbba3f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:44.876071 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a308af94-370a-46d7-bc30-1aeab5dbba3f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:44.876346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a308af94-370a-46d7-bc30-1aeab5dbba3f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:44.886009 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a308af94-370a-46d7-bc30-1aeab5dbba3f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:44.887464 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a308af94-370a-46d7-bc30-1aeab5dbba3f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:44.901492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-a308af94-370a-46d7-bc30-1aeab5dbba3f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:30:44.901492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:44.901492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:44.901492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:44.901492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:44.901492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:44.901492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:44.901492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:44.901492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:44.901492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:44.901492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:44.901492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:44.901492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:30:44.901492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:30:44.901492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:30:44.901492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:30:44.902614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:44.902614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:44.902614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:44.902614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:44.902614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:44.902614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:44.902614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:44.902614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:44.902614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:44.902614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:44.902614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:44.902614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:44.902614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:44.902614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:44.902614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:44.902614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:44.902614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:30:44.903720 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:30:44.903720 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:30:44.903720 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:30:44.903720 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 603/1215] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:30:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:30:44.908348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e2c4d1-b8c4-4f1f-b3dd-b24a5fb64426 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:44.967765 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:45.006951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e2c4d1-b8c4-4f1f-b3dd-b24a5fb64426 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:45.040945 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d38ae67-3861-4d8f-8838-faee79811efa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:45.041580 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d38ae67-3861-4d8f-8838-faee79811efa None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:45.041855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d38ae67-3861-4d8f-8838-faee79811efa None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:45.042126 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d38ae67-3861-4d8f-8838-faee79811efa None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:45.042784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d38ae67-3861-4d8f-8838-faee79811efa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:45.043086 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d38ae67-3861-4d8f-8838-faee79811efa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:45.044448 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d38ae67-3861-4d8f-8838-faee79811efa None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:45.055864 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 613/1216] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:30:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:45.061555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a308af94-370a-46d7-bc30-1aeab5dbba3f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:45.129715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:45.159919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a308af94-370a-46d7-bc30-1aeab5dbba3f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:45.193707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1d3628-30d5-495f-80e9-5c1d7e825075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:45.194337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1d3628-30d5-495f-80e9-5c1d7e825075 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:45.194633 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1d3628-30d5-495f-80e9-5c1d7e825075 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:45.194910 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1d3628-30d5-495f-80e9-5c1d7e825075 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:45.202461 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1d3628-30d5-495f-80e9-5c1d7e825075 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:45.204151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1d3628-30d5-495f-80e9-5c1d7e825075 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:45.215623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-de1d3628-30d5-495f-80e9-5c1d7e825075 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:30:45.215623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:45.215623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:45.215623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:45.215623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:45.215623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:45.215623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:45.215623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:45.215623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:45.215623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:45.215623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:45.215623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:45.215623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:30:45.215623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:30:45.215623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:30:45.217320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:30:45.217320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:45.217320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:45.217320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:45.217320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:45.217320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:45.217320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:45.217320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:45.217320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:45.217320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:45.217320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:45.217320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:45.217320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:45.217320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:45.217320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:45.217320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:45.217320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:45.219494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:30:45.219494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:30:45.219494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:30:45.219494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:30:45.219494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:30:45.219494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:30:45.219494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:30:45.219494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:30:45.219494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 604/1217] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:30:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:30:45.224883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d38ae67-3861-4d8f-8838-faee79811efa None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:45.322546 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d38ae67-3861-4d8f-8838-faee79811efa None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:45.344884 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:45.356655 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af7efec-1798-4b36-aa3f-2fd914824c99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:45.357388 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af7efec-1798-4b36-aa3f-2fd914824c99 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:45.357665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af7efec-1798-4b36-aa3f-2fd914824c99 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:45.357936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af7efec-1798-4b36-aa3f-2fd914824c99 None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:45.358624 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af7efec-1798-4b36-aa3f-2fd914824c99 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:45.358962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af7efec-1798-4b36-aa3f-2fd914824c99 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:45.360523 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af7efec-1798-4b36-aa3f-2fd914824c99 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:45.371609 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 614/1218] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:30:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:45.378057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1d3628-30d5-495f-80e9-5c1d7e825075 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:45.473741 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1d3628-30d5-495f-80e9-5c1d7e825075 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:45.501787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:45.507734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbcf120-33b9-49e5-8a95-1fc83f9b3b8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:45.508330 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbcf120-33b9-49e5-8a95-1fc83f9b3b8a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:45.508643 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbcf120-33b9-49e5-8a95-1fc83f9b3b8a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:45.508918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbcf120-33b9-49e5-8a95-1fc83f9b3b8a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:45.509622 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dbcf120-33b9-49e5-8a95-1fc83f9b3b8a None admin] RBAC: Adding query filter params (role.id=144dd855f74945548ea65db37acda1c1, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=49a3011c103b4e48904f876f81d27fba) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:45.509925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dbcf120-33b9-49e5-8a95-1fc83f9b3b8a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:45.511397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dbcf120-33b9-49e5-8a95-1fc83f9b3b8a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:45.516558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 605/1219] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:30:45 2022] GET /identity/v3/role_assignments?user.id=49a3011c103b4e48904f876f81d27fba&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=144dd855f74945548ea65db37acda1c1 => generated 259 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:47.636894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a9290c-1529-40aa-a915-69819f60e960 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:47.637441 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a9290c-1529-40aa-a915-69819f60e960 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:47.637441 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a9290c-1529-40aa-a915-69819f60e960 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:47.638803 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 615/1220] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:30:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:30:47.643940 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a1ceed-a9e1-4872-bec7-feefddbf9ab5 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:47.644318 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a1ceed-a9e1-4872-bec7-feefddbf9ab5 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:47.644485 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a1ceed-a9e1-4872-bec7-feefddbf9ab5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:47.663233 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-94a1ceed-a9e1-4872-bec7-feefddbf9ab5 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:30:47.664411 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-94a1ceed-a9e1-4872-bec7-feefddbf9ab5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:30:47.728107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 606/1221] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:30:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:30:48.000638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a9290c-1529-40aa-a915-69819f60e960 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:48.101585 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a9290c-1529-40aa-a915-69819f60e960 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:48.134709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e27bac9-7d0a-4976-a162-a23297efdc5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:48.135325 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e27bac9-7d0a-4976-a162-a23297efdc5f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:48.135600 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e27bac9-7d0a-4976-a162-a23297efdc5f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:48.135872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e27bac9-7d0a-4976-a162-a23297efdc5f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:48.146982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e27bac9-7d0a-4976-a162-a23297efdc5f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:30:48.152089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-1e27bac9-7d0a-4976-a162-a23297efdc5f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:30:48.152089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:48.152089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:48.152089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:48.152089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:48.152089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:48.152089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:48.152089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:48.152089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:48.152089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:48.152089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:48.152089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:48.152089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:30:48.152089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:30:48.152089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:30:48.152089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:30:48.153257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:30:48.153257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:30:48.153257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:30:48.153257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:30:48.153257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:48.153257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:48.153257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:48.153257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:48.153257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:48.153257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:48.153257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:48.153257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:48.153257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:48.153257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:48.153257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:48.153257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:48.154286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:48.154286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:48.154286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:48.154286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:48.154286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:30:48.154286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:30:48.154286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:30:48.154286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:30:48.154286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:30:48.154286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:30:48.154286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:30:48.154286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:30:48.154286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 616/1222] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:30:47 2022] GET /identity/v3/roles/service => generated 85 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:30:48.159318 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a1ceed-a9e1-4872-bec7-feefddbf9ab5 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:48.257867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a1ceed-a9e1-4872-bec7-feefddbf9ab5 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:48.280388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:48.291444 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1516c34b-db15-4b0d-8f4d-ac141ddeeb86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:48.292043 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1516c34b-db15-4b0d-8f4d-ac141ddeeb86 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:48.292406 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1516c34b-db15-4b0d-8f4d-ac141ddeeb86 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:48.292660 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1516c34b-db15-4b0d-8f4d-ac141ddeeb86 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:48.293302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1516c34b-db15-4b0d-8f4d-ac141ddeeb86 None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:48.293601 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1516c34b-db15-4b0d-8f4d-ac141ddeeb86 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:48.295052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1516c34b-db15-4b0d-8f4d-ac141ddeeb86 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:48.302614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 607/1223] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:30:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:48.308584 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e27bac9-7d0a-4976-a162-a23297efdc5f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:48.352384 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:48.406821 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e27bac9-7d0a-4976-a162-a23297efdc5f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:48.439441 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcc8781-a061-49b2-a385-102503a28537 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:48.440056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcc8781-a061-49b2-a385-102503a28537 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:48.440330 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcc8781-a061-49b2-a385-102503a28537 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:48.440598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcc8781-a061-49b2-a385-102503a28537 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:48.450789 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcc8781-a061-49b2-a385-102503a28537 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:48.452244 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcc8781-a061-49b2-a385-102503a28537 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:48.466965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-3fcc8781-a061-49b2-a385-102503a28537 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:30:48.466965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:48.466965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:48.466965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:48.466965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:48.466965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:48.466965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:48.466965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:48.466965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:48.466965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:48.466965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:48.466965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:48.466965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:30:48.466965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:30:48.466965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:30:48.466965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:30:48.468100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:48.468100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:48.468100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:48.468100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:48.468100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:48.468100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:48.468100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:48.468100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:48.468100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:48.468100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:48.468100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:48.468100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:48.468100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:48.468100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:48.468100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:48.468100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:48.468100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:30:48.469339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:30:48.469339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:30:48.469339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:30:48.469339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 617/1224] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:30:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:30:48.474039 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1516c34b-db15-4b0d-8f4d-ac141ddeeb86 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:48.570458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1516c34b-db15-4b0d-8f4d-ac141ddeeb86 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:48.602519 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e306860b-78d1-49ac-a860-59d2c593752e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:48.603147 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e306860b-78d1-49ac-a860-59d2c593752e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:48.603425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e306860b-78d1-49ac-a860-59d2c593752e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:48.603717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e306860b-78d1-49ac-a860-59d2c593752e None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:48.604342 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e306860b-78d1-49ac-a860-59d2c593752e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:48.604642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e306860b-78d1-49ac-a860-59d2c593752e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:48.605991 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e306860b-78d1-49ac-a860-59d2c593752e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:48.617256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 608/1225] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:30:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:48.622837 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcc8781-a061-49b2-a385-102503a28537 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:48.716929 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:48.723339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcc8781-a061-49b2-a385-102503a28537 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:48.756828 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1df1c0c-1d7f-41fe-a901-3cfcc433c060 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:48.757449 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1df1c0c-1d7f-41fe-a901-3cfcc433c060 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:48.757727 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1df1c0c-1d7f-41fe-a901-3cfcc433c060 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:48.757995 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1df1c0c-1d7f-41fe-a901-3cfcc433c060 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:48.764800 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1df1c0c-1d7f-41fe-a901-3cfcc433c060 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:48.766241 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1df1c0c-1d7f-41fe-a901-3cfcc433c060 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:48.777805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-f1df1c0c-1d7f-41fe-a901-3cfcc433c060 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 30 09:30:48.777805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:48.777805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:48.777805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:48.777805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:48.777805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:48.777805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:48.777805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:48.777805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:48.777805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:48.777805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:48.777805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:48.777805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:30:48.777805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:30:48.777805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:30:48.777805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:30:48.782773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:48.782773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:48.782773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:30:48.782773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:30:48.782773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:30:48.782773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:30:48.782773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:48.782773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:48.782773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:48.782773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:48.782773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:48.782773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:48.782773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:48.782773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:48.782773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:48.782773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:48.782773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:48.782773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:48.783968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:48.783968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:48.783968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:30:48.783968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:30:48.783968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:30:48.783968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:30:48.783968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:30:48.783968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:30:48.783968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 30 09:30:48.783968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:30:48.783968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 618/1226] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:30:48 2022] GET /identity/v3/users/placement => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:30:48.784751 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e306860b-78d1-49ac-a860-59d2c593752e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:48.830682 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:48.881702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e306860b-78d1-49ac-a860-59d2c593752e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:48.913544 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04972f2-7a96-42e7-be22-b9e3040f4d0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:48.914138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04972f2-7a96-42e7-be22-b9e3040f4d0e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:48.914420 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04972f2-7a96-42e7-be22-b9e3040f4d0e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:48.914718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04972f2-7a96-42e7-be22-b9e3040f4d0e None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:48.915430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04972f2-7a96-42e7-be22-b9e3040f4d0e None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:48.915733 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04972f2-7a96-42e7-be22-b9e3040f4d0e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:48.917154 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04972f2-7a96-42e7-be22-b9e3040f4d0e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:48.933981 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 609/1227] 10.222.0.39 () {56 vars in 1143 bytes} [Fri Sep 30 09:30:48 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:48.939897 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1df1c0c-1d7f-41fe-a901-3cfcc433c060 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:49.038604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1df1c0c-1d7f-41fe-a901-3cfcc433c060 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:49.072411 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65198c1d-bf5f-4795-bc88-b23d2c57d292 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:49.073032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65198c1d-bf5f-4795-bc88-b23d2c57d292 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:49.073308 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65198c1d-bf5f-4795-bc88-b23d2c57d292 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:49.073575 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65198c1d-bf5f-4795-bc88-b23d2c57d292 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:49.084076 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65198c1d-bf5f-4795-bc88-b23d2c57d292 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:49.085492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65198c1d-bf5f-4795-bc88-b23d2c57d292 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:49.100129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-65198c1d-bf5f-4795-bc88-b23d2c57d292 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:30:49.100129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:49.100129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:49.100129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:49.100129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:49.100129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:49.100129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:49.100129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:49.100129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:49.100129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:49.100129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:49.100129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:49.100129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:30:49.100129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:30:49.100129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:30:49.100129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:30:49.101296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:49.101296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:49.101296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:49.101296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:49.101296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:49.101296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:49.101296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:49.101296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:49.101296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:49.101296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:49.101296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:49.101296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:49.101296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:49.101296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:49.101296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:49.101296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:49.101296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:30:49.102443 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:30:49.102443 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:30:49.102443 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:30:49.102443 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 619/1228] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:30:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:30:49.107032 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04972f2-7a96-42e7-be22-b9e3040f4d0e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:49.203926 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04972f2-7a96-42e7-be22-b9e3040f4d0e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:49.206308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:49.237243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d7845e-19ae-4f58-9856-afd6a447160d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:49.237870 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d7845e-19ae-4f58-9856-afd6a447160d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:49.238147 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d7845e-19ae-4f58-9856-afd6a447160d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:49.238467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d7845e-19ae-4f58-9856-afd6a447160d None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:49.239708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d7845e-19ae-4f58-9856-afd6a447160d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:49.240007 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d7845e-19ae-4f58-9856-afd6a447160d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:49.241354 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d7845e-19ae-4f58-9856-afd6a447160d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:49.252234 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 610/1229] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:30:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:49.258026 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65198c1d-bf5f-4795-bc88-b23d2c57d292 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:49.267739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:49.361530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65198c1d-bf5f-4795-bc88-b23d2c57d292 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:49.395767 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed3be61-8163-4fde-bb7f-62327d0f54df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:49.396393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed3be61-8163-4fde-bb7f-62327d0f54df None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:49.396699 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed3be61-8163-4fde-bb7f-62327d0f54df None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:49.396967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed3be61-8163-4fde-bb7f-62327d0f54df None admin] PATH_INFO: `/v3/projects/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:49.403576 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed3be61-8163-4fde-bb7f-62327d0f54df None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:49.405101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed3be61-8163-4fde-bb7f-62327d0f54df None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:49.416739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-eed3be61-8163-4fde-bb7f-62327d0f54df None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:30:49.416739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:49.416739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:49.416739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:49.416739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:49.416739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:49.416739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:49.416739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:49.416739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:49.416739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:49.416739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:49.416739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:49.416739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:30:49.416739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:30:49.416739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:30:49.417679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:30:49.417679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:49.417679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:49.417679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:49.417679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:49.417679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:49.417679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:49.417679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:49.417679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:49.417679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:49.417679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:49.417679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:49.417679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:49.417679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:49.417679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:49.417679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:49.417679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:49.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:30:49.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:30:49.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:30:49.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:30:49.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:30:49.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:30:49.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:30:49.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:30:49.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 620/1230] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:30:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:30:49.423695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d7845e-19ae-4f58-9856-afd6a447160d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:49.522046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d7845e-19ae-4f58-9856-afd6a447160d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:49.554311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9eb84c-119e-44f5-9c28-87b434b67621 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:49.554929 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9eb84c-119e-44f5-9c28-87b434b67621 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:49.555205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9eb84c-119e-44f5-9c28-87b434b67621 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:49.555470 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9eb84c-119e-44f5-9c28-87b434b67621 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:49.556141 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9eb84c-119e-44f5-9c28-87b434b67621 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:49.556440 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9eb84c-119e-44f5-9c28-87b434b67621 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:49.557857 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9eb84c-119e-44f5-9c28-87b434b67621 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:49.569428 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 611/1231] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:30:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:49.575434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed3be61-8163-4fde-bb7f-62327d0f54df None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:49.640962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:49.675209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed3be61-8163-4fde-bb7f-62327d0f54df None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:49.707459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773c6ff7-8d89-4c8a-baf3-168c434b028e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:49.708599 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773c6ff7-8d89-4c8a-baf3-168c434b028e None admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:49.708874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773c6ff7-8d89-4c8a-baf3-168c434b028e None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:49.709140 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773c6ff7-8d89-4c8a-baf3-168c434b028e None admin] PATH_INFO: `/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/49a3011c103b4e48904f876f81d27fba/roles/144dd855f74945548ea65db37acda1c1` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:49.723052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773c6ff7-8d89-4c8a-baf3-168c434b028e None admin] RBAC: Authorizing `identity:create_grant(project_id=c3802ced1fce4997a016d41eca843ef8, user_id=49a3011c103b4e48904f876f81d27fba, role_id=144dd855f74945548ea65db37acda1c1)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:49.725069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773c6ff7-8d89-4c8a-baf3-168c434b028e None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:49.853326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 621/1232] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:30:49 2022] PUT /identity/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/49a3011c103b4e48904f876f81d27fba/roles/144dd855f74945548ea65db37acda1c1 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:30:51.995529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0f64cc-2cbb-42f9-a89d-fdcd37234875 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:51.996103 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0f64cc-2cbb-42f9-a89d-fdcd37234875 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:51.996103 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0f64cc-2cbb-42f9-a89d-fdcd37234875 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:51.997545 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 612/1233] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:30:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:30:52.003184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f96f7c2-ee12-44ca-a521-7fad8e12995e None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:52.005021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f96f7c2-ee12-44ca-a521-7fad8e12995e None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:52.005021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f96f7c2-ee12-44ca-a521-7fad8e12995e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:52.022565 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-7f96f7c2-ee12-44ca-a521-7fad8e12995e None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:30:52.023772 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7f96f7c2-ee12-44ca-a521-7fad8e12995e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:30:52.088275 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 622/1234] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:30:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:30:52.373727 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0f64cc-2cbb-42f9-a89d-fdcd37234875 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:52.398413 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:52.471687 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0f64cc-2cbb-42f9-a89d-fdcd37234875 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:52.507869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7898f3ed-6c59-4caa-8fe7-0b9f3d16dc91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:52.508524 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7898f3ed-6c59-4caa-8fe7-0b9f3d16dc91 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:52.508805 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7898f3ed-6c59-4caa-8fe7-0b9f3d16dc91 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:52.509075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7898f3ed-6c59-4caa-8fe7-0b9f3d16dc91 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:52.519024 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7898f3ed-6c59-4caa-8fe7-0b9f3d16dc91 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:30:52.524196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-7898f3ed-6c59-4caa-8fe7-0b9f3d16dc91 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:30:52.524196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:52.524196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:52.524196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:52.524196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:52.524196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:52.524196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:52.524196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:52.524196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:52.524196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:52.524196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:52.524196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:52.524196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:30:52.524196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:30:52.524196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:30:52.524196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:30:52.525346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:30:52.525346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:30:52.525346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:30:52.525346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:30:52.525346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:52.525346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:52.525346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:52.525346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:52.525346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:52.525346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:52.525346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:52.525346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:52.525346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:52.525346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:52.525346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:52.525346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:52.526355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:52.526355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:52.526355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:52.526355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:52.526355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:30:52.526355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:30:52.526355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:30:52.526355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:30:52.526355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:30:52.526355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:30:52.526355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 30 09:30:52.526355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:30:52.526355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 613/1235] 10.222.0.39 () {56 vars in 1102 bytes} [Fri Sep 30 09:30:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:30:52.531272 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f96f7c2-ee12-44ca-a521-7fad8e12995e None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:52.575610 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:52.626845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f96f7c2-ee12-44ca-a521-7fad8e12995e None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:52.659216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca77521-9313-4dbe-8def-03fe44e8b4be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:52.659813 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca77521-9313-4dbe-8def-03fe44e8b4be None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:52.660094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca77521-9313-4dbe-8def-03fe44e8b4be None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:52.660365 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca77521-9313-4dbe-8def-03fe44e8b4be None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:52.661552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca77521-9313-4dbe-8def-03fe44e8b4be None admin] RBAC: Adding query filter params (name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:52.661849 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca77521-9313-4dbe-8def-03fe44e8b4be None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:52.663282 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca77521-9313-4dbe-8def-03fe44e8b4be None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:52.670938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 623/1236] 10.222.0.39 () {56 vars in 1103 bytes} [Fri Sep 30 09:30:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:52.678513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7898f3ed-6c59-4caa-8fe7-0b9f3d16dc91 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:52.762029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:52.777755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7898f3ed-6c59-4caa-8fe7-0b9f3d16dc91 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:52.810613 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0e1487-729a-4719-9aff-a03e48acdc5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:52.811230 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0e1487-729a-4719-9aff-a03e48acdc5e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:52.811505 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0e1487-729a-4719-9aff-a03e48acdc5e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:52.811772 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0e1487-729a-4719-9aff-a03e48acdc5e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:52.821250 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0e1487-729a-4719-9aff-a03e48acdc5e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:52.822701 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0e1487-729a-4719-9aff-a03e48acdc5e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:52.837200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-7f0e1487-729a-4719-9aff-a03e48acdc5e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:30:52.837200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:52.837200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:52.837200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:52.837200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:52.837200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:52.837200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:52.837200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:52.837200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:52.837200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:52.837200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:52.837200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:52.837200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:30:52.837200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:30:52.837200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:30:52.837200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:30:52.842546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:52.842546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:52.842546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:52.842546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:52.842546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:52.842546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:52.842546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:52.842546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:52.842546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:52.842546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:52.842546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:52.842546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:52.842546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:52.842546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:52.842546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:52.842546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:52.842546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:30:52.843676 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:30:52.843676 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:30:52.843676 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:30:52.843676 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 614/1237] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:30:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:30:52.844210 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca77521-9313-4dbe-8def-03fe44e8b4be None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:52.943962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca77521-9313-4dbe-8def-03fe44e8b4be None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:52.953694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:52.982425 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2872d0-9beb-4cd4-9529-302b2d11482f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:52.983089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2872d0-9beb-4cd4-9529-302b2d11482f None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:52.983358 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2872d0-9beb-4cd4-9529-302b2d11482f None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:52.983637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2872d0-9beb-4cd4-9529-302b2d11482f None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:52.984274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2872d0-9beb-4cd4-9529-302b2d11482f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:52.984594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2872d0-9beb-4cd4-9529-302b2d11482f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:52.986091 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2872d0-9beb-4cd4-9529-302b2d11482f None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:52.998179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 624/1238] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:30:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:53.004874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0e1487-729a-4719-9aff-a03e48acdc5e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:53.103401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0e1487-729a-4719-9aff-a03e48acdc5e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:53.135920 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d06ee5-37d9-4100-af6d-acf416ed78c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:53.136531 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d06ee5-37d9-4100-af6d-acf416ed78c3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:53.136807 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d06ee5-37d9-4100-af6d-acf416ed78c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:53.137080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d06ee5-37d9-4100-af6d-acf416ed78c3 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:53.139693 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:53.144481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d06ee5-37d9-4100-af6d-acf416ed78c3 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:53.146544 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d06ee5-37d9-4100-af6d-acf416ed78c3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:53.157288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-15d06ee5-37d9-4100-af6d-acf416ed78c3 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 30 09:30:53.157288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:53.157288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:53.157288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:53.157288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:53.157288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:53.157288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:53.157288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:53.157288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:53.157288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:53.157288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:53.157288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:53.157288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:30:53.157288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:30:53.157288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:30:53.157288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:30:53.158517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:53.158517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:53.158517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:30:53.158517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:30:53.158517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:30:53.158517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:30:53.158517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:53.158517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:53.158517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:53.158517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:53.158517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:53.158517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:53.158517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:53.158517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:53.158517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:53.158517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:53.158517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:53.158517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:53.159757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:53.159757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:53.159757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:30:53.159757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:30:53.159757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:30:53.159757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:30:53.159757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:30:53.159757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:30:53.159757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 30 09:30:53.159757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:30:53.159757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 615/1239] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:30:53 2022] GET /identity/v3/users/placement => generated 87 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:30:53.164547 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2872d0-9beb-4cd4-9529-302b2d11482f None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:53.262987 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2872d0-9beb-4cd4-9529-302b2d11482f None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:53.295669 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41eecb5-1798-4fbc-81bd-39967d8f522e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:53.296262 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41eecb5-1798-4fbc-81bd-39967d8f522e None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:53.296543 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41eecb5-1798-4fbc-81bd-39967d8f522e None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:53.296816 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41eecb5-1798-4fbc-81bd-39967d8f522e None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:53.297527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41eecb5-1798-4fbc-81bd-39967d8f522e None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:53.297832 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41eecb5-1798-4fbc-81bd-39967d8f522e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:53.299308 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41eecb5-1798-4fbc-81bd-39967d8f522e None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:53.316287 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 625/1240] 10.222.0.39 () {56 vars in 1143 bytes} [Fri Sep 30 09:30:53 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:53.322760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d06ee5-37d9-4100-af6d-acf416ed78c3 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:53.419208 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d06ee5-37d9-4100-af6d-acf416ed78c3 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:53.452731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0c4421-c62e-4128-9b1c-2b62f8645216 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:53.453243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0c4421-c62e-4128-9b1c-2b62f8645216 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:53.453589 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0c4421-c62e-4128-9b1c-2b62f8645216 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:53.453858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0c4421-c62e-4128-9b1c-2b62f8645216 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:53.513252 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:53.519743 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0c4421-c62e-4128-9b1c-2b62f8645216 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:53.521169 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0c4421-c62e-4128-9b1c-2b62f8645216 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:53.617928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 616/1241] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:30:53 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 30 09:30:53.623762 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41eecb5-1798-4fbc-81bd-39967d8f522e None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:53.650289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:53.719519 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41eecb5-1798-4fbc-81bd-39967d8f522e None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:53.754754 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a34da0-3958-44c0-9171-1bb50b70bb3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:53.755363 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a34da0-3958-44c0-9171-1bb50b70bb3c None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:53.755642 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a34da0-3958-44c0-9171-1bb50b70bb3c None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:53.755905 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a34da0-3958-44c0-9171-1bb50b70bb3c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:53.765538 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a34da0-3958-44c0-9171-1bb50b70bb3c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:53.766994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a34da0-3958-44c0-9171-1bb50b70bb3c None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:53.780907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-c5a34da0-3958-44c0-9171-1bb50b70bb3c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:30:53.780907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:53.780907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:53.780907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:53.780907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:53.780907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:53.780907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:53.780907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:53.780907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:53.780907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:53.780907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:53.780907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:53.780907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:30:53.780907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:30:53.780907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:30:53.780907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:30:53.782101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:53.782101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:53.782101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:53.782101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:53.782101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:53.782101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:53.782101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:53.782101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:53.782101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:53.782101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:53.782101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:53.782101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:53.782101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:53.782101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:53.782101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:53.782101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:53.782101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:30:53.788213 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:30:53.788213 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:30:53.788213 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:30:53.788213 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 626/1242] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:30:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:30:53.788502 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0c4421-c62e-4128-9b1c-2b62f8645216 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:53.878039 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:53.883561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0c4421-c62e-4128-9b1c-2b62f8645216 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:53.915309 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88bb7b6-70ca-4ef8-b6b1-7772a83aef3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:53.915903 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88bb7b6-70ca-4ef8-b6b1-7772a83aef3a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:53.916179 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88bb7b6-70ca-4ef8-b6b1-7772a83aef3a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:53.916448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88bb7b6-70ca-4ef8-b6b1-7772a83aef3a None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:53.917113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88bb7b6-70ca-4ef8-b6b1-7772a83aef3a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:53.917409 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88bb7b6-70ca-4ef8-b6b1-7772a83aef3a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:53.918776 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88bb7b6-70ca-4ef8-b6b1-7772a83aef3a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:53.929942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 617/1243] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:30:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:53.935533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a34da0-3958-44c0-9171-1bb50b70bb3c None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:54.014613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:54.035988 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a34da0-3958-44c0-9171-1bb50b70bb3c None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:54.070696 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a95a176-e88c-4237-ba0c-66b4036d104a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:54.071320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a95a176-e88c-4237-ba0c-66b4036d104a None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:54.071601 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a95a176-e88c-4237-ba0c-66b4036d104a None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:54.071871 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a95a176-e88c-4237-ba0c-66b4036d104a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:54.078440 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a95a176-e88c-4237-ba0c-66b4036d104a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:54.079919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a95a176-e88c-4237-ba0c-66b4036d104a None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:54.091579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-2a95a176-e88c-4237-ba0c-66b4036d104a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:30:54.091579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:54.091579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:54.091579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:54.091579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:54.091579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:54.091579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:54.091579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:54.091579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:54.091579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:54.091579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:54.091579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:54.091579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:30:54.091579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:30:54.091579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:30:54.092628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:30:54.092628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:54.092628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:54.092628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:54.092628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:54.092628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:54.092628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:54.092628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:54.092628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:54.092628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:54.092628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:54.092628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:54.092628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:54.092628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:54.092628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:54.092628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:54.092628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:54.094014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:30:54.094014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:30:54.094014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:30:54.094014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:30:54.094014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:30:54.094014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:30:54.094014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:30:54.094014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:30:54.094014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 627/1244] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:30:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:30:54.099222 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88bb7b6-70ca-4ef8-b6b1-7772a83aef3a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:54.199532 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88bb7b6-70ca-4ef8-b6b1-7772a83aef3a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:54.233210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f498872a-0116-48f9-ad7f-ad780885090e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:54.233807 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f498872a-0116-48f9-ad7f-ad780885090e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:54.234084 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f498872a-0116-48f9-ad7f-ad780885090e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:54.234353 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f498872a-0116-48f9-ad7f-ad780885090e None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:54.235051 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f498872a-0116-48f9-ad7f-ad780885090e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:54.235354 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f498872a-0116-48f9-ad7f-ad780885090e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:54.236800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f498872a-0116-48f9-ad7f-ad780885090e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:54.248641 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 618/1245] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:30:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:54.255003 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a95a176-e88c-4237-ba0c-66b4036d104a None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:54.353141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a95a176-e88c-4237-ba0c-66b4036d104a None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:54.385757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac6a147-7969-4216-9754-a6888fe345db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:54.386339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac6a147-7969-4216-9754-a6888fe345db None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:54.386636 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac6a147-7969-4216-9754-a6888fe345db None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:54.386909 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac6a147-7969-4216-9754-a6888fe345db None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:54.387610 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac6a147-7969-4216-9754-a6888fe345db None admin] RBAC: Adding query filter params (role.id=144dd855f74945548ea65db37acda1c1, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=49a3011c103b4e48904f876f81d27fba) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:54.387953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac6a147-7969-4216-9754-a6888fe345db None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:54.389409 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac6a147-7969-4216-9754-a6888fe345db None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:54.395551 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 628/1246] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:30:54 2022] GET /identity/v3/role_assignments?user.id=49a3011c103b4e48904f876f81d27fba&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=144dd855f74945548ea65db37acda1c1 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:56.627332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a07583a-b398-4c72-9c0f-1e9b3dfe689e None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:56.627861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a07583a-b398-4c72-9c0f-1e9b3dfe689e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:56.627861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a07583a-b398-4c72-9c0f-1e9b3dfe689e None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:56.629739 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 619/1247] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:30:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:30:56.635350 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a20c24-caf9-4a04-b5b1-974b54b59ed4 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:56.635772 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a20c24-caf9-4a04-b5b1-974b54b59ed4 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:56.635898 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a20c24-caf9-4a04-b5b1-974b54b59ed4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:56.642226 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:56.655229 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-c1a20c24-caf9-4a04-b5b1-974b54b59ed4 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:30:56.656583 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c1a20c24-caf9-4a04-b5b1-974b54b59ed4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:30:56.720488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 629/1248] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:30:56 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:30:56.988076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a07583a-b398-4c72-9c0f-1e9b3dfe689e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:57.007265 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:57.088068 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a07583a-b398-4c72-9c0f-1e9b3dfe689e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:57.120603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40b0172-1e08-4e62-8304-2400e3946ab0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:57.121209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40b0172-1e08-4e62-8304-2400e3946ab0 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:57.121485 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40b0172-1e08-4e62-8304-2400e3946ab0 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:57.121754 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40b0172-1e08-4e62-8304-2400e3946ab0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:57.132450 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40b0172-1e08-4e62-8304-2400e3946ab0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:30:57.137184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-a40b0172-1e08-4e62-8304-2400e3946ab0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:30:57.137184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:57.137184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:57.137184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:57.137184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:57.137184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:57.137184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:57.137184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:57.137184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:57.137184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:57.137184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:57.137184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:57.137184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:30:57.137184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:30:57.137184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:30:57.137184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:30:57.138293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:30:57.138293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:30:57.138293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:30:57.138293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:30:57.138293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:57.138293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:57.138293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:57.138293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:57.138293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:57.138293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:57.138293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:57.138293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:57.138293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:57.138293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:57.138293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:57.138293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:57.139209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:57.139209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:57.139209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:57.139209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:57.139209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:30:57.139209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:30:57.139209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:30:57.139209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:30:57.139209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:30:57.139209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:30:57.139209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:30:57.139209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:30:57.139209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 620/1249] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:30:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:30:57.144529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a20c24-caf9-4a04-b5b1-974b54b59ed4 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:57.241911 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a20c24-caf9-4a04-b5b1-974b54b59ed4 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:57.270913 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:57.275670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed261c74-33be-4ac6-a1a1-6c580ed04d8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:57.276261 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed261c74-33be-4ac6-a1a1-6c580ed04d8b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:57.276536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed261c74-33be-4ac6-a1a1-6c580ed04d8b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:57.276801 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed261c74-33be-4ac6-a1a1-6c580ed04d8b None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:57.277448 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed261c74-33be-4ac6-a1a1-6c580ed04d8b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:57.277744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed261c74-33be-4ac6-a1a1-6c580ed04d8b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:57.279132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed261c74-33be-4ac6-a1a1-6c580ed04d8b None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:57.287229 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 630/1250] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:30:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:57.293200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40b0172-1e08-4e62-8304-2400e3946ab0 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:57.369434 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:57.391249 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40b0172-1e08-4e62-8304-2400e3946ab0 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:57.423340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca08c1c1-6373-4309-9283-7ba69be666ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:57.423961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca08c1c1-6373-4309-9283-7ba69be666ef None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:57.424241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca08c1c1-6373-4309-9283-7ba69be666ef None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:57.424514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca08c1c1-6373-4309-9283-7ba69be666ef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:57.434538 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca08c1c1-6373-4309-9283-7ba69be666ef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:57.436011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca08c1c1-6373-4309-9283-7ba69be666ef None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:57.449885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-ca08c1c1-6373-4309-9283-7ba69be666ef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:30:57.449885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:57.449885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:57.449885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:57.449885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:57.449885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:57.449885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:57.449885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:57.449885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:57.449885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:57.449885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:57.449885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:57.449885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:30:57.449885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:30:57.449885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:30:57.449885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:30:57.451100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:57.451100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:57.451100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:57.451100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:57.451100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:57.451100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:57.451100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:57.451100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:57.451100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:57.451100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:57.451100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:57.451100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:57.451100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:57.451100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:57.451100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:57.451100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:57.451100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:30:57.452037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:30:57.452037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:30:57.452037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:30:57.452037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 621/1251] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:30:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:30:57.456972 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed261c74-33be-4ac6-a1a1-6c580ed04d8b None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:57.556789 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed261c74-33be-4ac6-a1a1-6c580ed04d8b None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:57.588716 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de754eb-d089-419c-8928-28476139fc2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:57.589309 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de754eb-d089-419c-8928-28476139fc2b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:57.589587 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de754eb-d089-419c-8928-28476139fc2b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:57.589857 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de754eb-d089-419c-8928-28476139fc2b None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:57.590483 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de754eb-d089-419c-8928-28476139fc2b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:57.590813 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de754eb-d089-419c-8928-28476139fc2b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:57.592181 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de754eb-d089-419c-8928-28476139fc2b None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:57.604202 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 631/1252] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:30:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:57.610060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca08c1c1-6373-4309-9283-7ba69be666ef None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:57.712739 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca08c1c1-6373-4309-9283-7ba69be666ef None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:57.737639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:57.745743 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f08a480-b335-447b-9880-634c5e748f63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:57.746365 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f08a480-b335-447b-9880-634c5e748f63 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:57.746672 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f08a480-b335-447b-9880-634c5e748f63 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:57.746945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f08a480-b335-447b-9880-634c5e748f63 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:57.754460 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f08a480-b335-447b-9880-634c5e748f63 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:57.757569 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f08a480-b335-447b-9880-634c5e748f63 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:57.766787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-4f08a480-b335-447b-9880-634c5e748f63 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 30 09:30:57.766787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:57.766787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:57.766787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:57.766787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:57.766787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:57.766787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:57.766787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:57.766787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:57.766787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:57.766787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:57.766787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:57.766787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:30:57.766787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:30:57.766787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:30:57.766787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:30:57.767844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:57.767844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:57.767844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:30:57.767844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:30:57.767844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:30:57.767844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:30:57.767844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:57.767844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:57.767844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:57.767844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:57.767844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:57.767844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:57.767844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:57.767844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:57.767844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:57.767844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:57.767844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:57.767844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:57.768817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:57.768817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:57.768817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:30:57.768817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:30:57.768817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:30:57.768817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:30:57.768817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:30:57.768817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:30:57.768817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 30 09:30:57.768817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:30:57.768817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 622/1253] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:30:57 2022] GET /identity/v3/users/placement => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:30:57.774414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de754eb-d089-419c-8928-28476139fc2b None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:57.830425 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:57.873653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de754eb-d089-419c-8928-28476139fc2b None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:57.907688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb40bfd-47b4-4eb1-8d15-36d9ec10044d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:57.908288 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb40bfd-47b4-4eb1-8d15-36d9ec10044d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:57.908566 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb40bfd-47b4-4eb1-8d15-36d9ec10044d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:57.908836 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb40bfd-47b4-4eb1-8d15-36d9ec10044d None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:57.909546 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb40bfd-47b4-4eb1-8d15-36d9ec10044d None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:57.909847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb40bfd-47b4-4eb1-8d15-36d9ec10044d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:57.911919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb40bfd-47b4-4eb1-8d15-36d9ec10044d None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:57.928598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 632/1254] 10.222.0.39 () {56 vars in 1143 bytes} [Fri Sep 30 09:30:57 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:57.934690 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f08a480-b335-447b-9880-634c5e748f63 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:58.031491 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f08a480-b335-447b-9880-634c5e748f63 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:58.065028 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe09f7a-46fa-4640-b10a-69976bbeb718 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:58.065633 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe09f7a-46fa-4640-b10a-69976bbeb718 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:58.065914 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe09f7a-46fa-4640-b10a-69976bbeb718 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:58.066188 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe09f7a-46fa-4640-b10a-69976bbeb718 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:58.120609 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:58.133334 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe09f7a-46fa-4640-b10a-69976bbeb718 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:58.134835 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe09f7a-46fa-4640-b10a-69976bbeb718 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:58.230158 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 623/1255] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:30:57 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 30 09:30:58.236017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb40bfd-47b4-4eb1-8d15-36d9ec10044d None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:58.336943 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb40bfd-47b4-4eb1-8d15-36d9ec10044d None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:58.351098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:58.371678 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d107d61-8711-4889-b7b9-f88724278aec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:58.372234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d107d61-8711-4889-b7b9-f88724278aec None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:58.372514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d107d61-8711-4889-b7b9-f88724278aec None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:58.372781 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d107d61-8711-4889-b7b9-f88724278aec None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:58.382489 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d107d61-8711-4889-b7b9-f88724278aec None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:58.389363 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d107d61-8711-4889-b7b9-f88724278aec None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:58.397799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-2d107d61-8711-4889-b7b9-f88724278aec None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:30:58.397799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:58.397799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:58.397799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:58.397799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:58.397799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:58.397799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:58.397799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:58.397799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:58.397799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:58.397799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:58.397799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:58.397799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:30:58.397799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:30:58.397799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:30:58.397799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:30:58.398974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:58.398974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:58.398974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:58.398974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:58.398974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:58.398974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:58.398974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:58.398974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:58.398974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:58.398974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:58.398974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:58.398974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:58.398974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:58.398974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:58.398974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:58.398974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:58.398974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:30:58.400069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:30:58.400069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:30:58.400069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:30:58.400069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 633/1256] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:30:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:30:58.404611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe09f7a-46fa-4640-b10a-69976bbeb718 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:58.488808 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:58.500280 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe09f7a-46fa-4640-b10a-69976bbeb718 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:58.532570 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b7acfb-5124-42cc-8b31-3ce1d269e7fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:58.533138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b7acfb-5124-42cc-8b31-3ce1d269e7fd None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:58.533421 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b7acfb-5124-42cc-8b31-3ce1d269e7fd None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:58.533689 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b7acfb-5124-42cc-8b31-3ce1d269e7fd None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:58.534314 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b7acfb-5124-42cc-8b31-3ce1d269e7fd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:58.534634 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b7acfb-5124-42cc-8b31-3ce1d269e7fd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:58.536616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b7acfb-5124-42cc-8b31-3ce1d269e7fd None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:58.547330 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 624/1257] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:30:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:58.553082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d107d61-8711-4889-b7b9-f88724278aec None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:58.653178 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d107d61-8711-4889-b7b9-f88724278aec None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:58.688953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f64fd4-2e81-49f7-9301-45a609417177 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:58.689573 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f64fd4-2e81-49f7-9301-45a609417177 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:58.689854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f64fd4-2e81-49f7-9301-45a609417177 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:58.690128 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f64fd4-2e81-49f7-9301-45a609417177 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:58.697463 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f64fd4-2e81-49f7-9301-45a609417177 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:58.699102 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f64fd4-2e81-49f7-9301-45a609417177 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:58.710090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-46f64fd4-2e81-49f7-9301-45a609417177 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:30:58.710090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:30:58.710090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:30:58.710090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:30:58.710090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:30:58.710090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:30:58.710090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:30:58.710090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:30:58.710090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:30:58.710090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:30:58.710090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:30:58.710090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:30:58.710090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:30:58.710090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:30:58.710090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:30:58.711211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:30:58.711211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:30:58.711211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:30:58.711211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:30:58.711211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:30:58.711211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:30:58.711211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:30:58.711211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:30:58.711211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:30:58.711211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:30:58.711211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:30:58.711211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:30:58.711211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:30:58.711211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:30:58.711211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:30:58.711211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:30:58.711211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:30:58.712337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:30:58.712337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:30:58.712337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:30:58.712337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:30:58.712337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:30:58.712337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:30:58.712337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:30:58.712337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:30:58.712337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 634/1258] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:30:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:30:58.717533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b7acfb-5124-42cc-8b31-3ce1d269e7fd None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:58.815098 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b7acfb-5124-42cc-8b31-3ce1d269e7fd None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:58.848524 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270ea9d5-6f5b-4244-b10c-56a09c904c81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:58.849115 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270ea9d5-6f5b-4244-b10c-56a09c904c81 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:58.849394 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270ea9d5-6f5b-4244-b10c-56a09c904c81 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:58.849667 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270ea9d5-6f5b-4244-b10c-56a09c904c81 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:58.850342 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270ea9d5-6f5b-4244-b10c-56a09c904c81 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:58.850705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270ea9d5-6f5b-4244-b10c-56a09c904c81 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:58.852158 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270ea9d5-6f5b-4244-b10c-56a09c904c81 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:58.863816 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 625/1259] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:30:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:30:58.869850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f64fd4-2e81-49f7-9301-45a609417177 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:30:58.895855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:30:58.969357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f64fd4-2e81-49f7-9301-45a609417177 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:30:59.001513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178f0939-7bf2-47ef-916c-feb4c90f3fc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:30:59.002112 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178f0939-7bf2-47ef-916c-feb4c90f3fc5 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:30:59.002367 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178f0939-7bf2-47ef-916c-feb4c90f3fc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:30:59.002657 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178f0939-7bf2-47ef-916c-feb4c90f3fc5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:30:59.003360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178f0939-7bf2-47ef-916c-feb4c90f3fc5 None admin] RBAC: Adding query filter params (role.id=e874c311084c4c9fb7691bce461d319b, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=49a3011c103b4e48904f876f81d27fba) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:30:59.004171 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178f0939-7bf2-47ef-916c-feb4c90f3fc5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:30:59.005647 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178f0939-7bf2-47ef-916c-feb4c90f3fc5 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:30:59.010950 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 635/1260] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:30:58 2022] GET /identity/v3/role_assignments?user.id=49a3011c103b4e48904f876f81d27fba&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=e874c311084c4c9fb7691bce461d319b => generated 259 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:31:01.120824 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5105808-5350-4d40-b6d2-6121bcf7b1f7 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:01.121439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5105808-5350-4d40-b6d2-6121bcf7b1f7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:01.121439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5105808-5350-4d40-b6d2-6121bcf7b1f7 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:01.122765 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 626/1261] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:31:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:31:01.128281 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2e54cb-4cfa-4513-9598-f97b9fd62fa9 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:01.128707 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2e54cb-4cfa-4513-9598-f97b9fd62fa9 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:01.128816 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2e54cb-4cfa-4513-9598-f97b9fd62fa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:01.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-8e2e54cb-4cfa-4513-9598-f97b9fd62fa9 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:31:01.151258 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-8e2e54cb-4cfa-4513-9598-f97b9fd62fa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:31:01.216550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 636/1262] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:31:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:31:01.483944 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5105808-5350-4d40-b6d2-6121bcf7b1f7 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:01.495387 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:01.583892 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5105808-5350-4d40-b6d2-6121bcf7b1f7 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:01.616877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300f606a-c138-4add-a601-ee517b716d25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:01.617501 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300f606a-c138-4add-a601-ee517b716d25 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:01.617780 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300f606a-c138-4add-a601-ee517b716d25 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:01.618053 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300f606a-c138-4add-a601-ee517b716d25 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:01.628215 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300f606a-c138-4add-a601-ee517b716d25 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:31:01.632958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-300f606a-c138-4add-a601-ee517b716d25 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:31:01.632958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:31:01.632958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:31:01.632958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:31:01.632958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:31:01.632958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:31:01.632958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:31:01.632958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:31:01.632958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:31:01.632958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:31:01.632958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:31:01.632958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:31:01.632958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:31:01.632958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:31:01.632958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:31:01.632958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:31:01.634058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:31:01.634058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:31:01.634058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:31:01.634058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:31:01.634058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:31:01.634058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:31:01.634058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:31:01.634058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:31:01.634058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:31:01.634058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:31:01.634058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:31:01.634058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:31:01.634058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:31:01.634058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:31:01.634058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:31:01.634058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:31:01.635046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:31:01.635046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:31:01.635046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:31:01.635046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:31:01.635046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:31:01.635046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:31:01.635046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:31:01.635046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:31:01.635046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:31:01.635046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:31:01.635046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:31:01.635046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:31:01.635046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 627/1263] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:31:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:31:01.640795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2e54cb-4cfa-4513-9598-f97b9fd62fa9 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:01.659068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:01.739165 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2e54cb-4cfa-4513-9598-f97b9fd62fa9 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:01.772590 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ceaddd-35f2-4b7c-8c20-ac13ce60c8d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:01.773179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ceaddd-35f2-4b7c-8c20-ac13ce60c8d6 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:01.773455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ceaddd-35f2-4b7c-8c20-ac13ce60c8d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:01.773726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ceaddd-35f2-4b7c-8c20-ac13ce60c8d6 None admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:01.774372 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ceaddd-35f2-4b7c-8c20-ac13ce60c8d6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:31:01.774695 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ceaddd-35f2-4b7c-8c20-ac13ce60c8d6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:01.776051 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ceaddd-35f2-4b7c-8c20-ac13ce60c8d6 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:01.784004 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 637/1264] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:31:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:31:01.790230 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300f606a-c138-4add-a601-ee517b716d25 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:01.859099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:01.891015 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300f606a-c138-4add-a601-ee517b716d25 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:01.922744 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ec05fb-d12b-43e0-872c-1616303571fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:01.923358 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ec05fb-d12b-43e0-872c-1616303571fb None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:01.923635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ec05fb-d12b-43e0-872c-1616303571fb None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:01.923905 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ec05fb-d12b-43e0-872c-1616303571fb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:01.933922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ec05fb-d12b-43e0-872c-1616303571fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:01.935392 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ec05fb-d12b-43e0-872c-1616303571fb None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:01.949718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-b0ec05fb-d12b-43e0-872c-1616303571fb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:31:01.949718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:31:01.949718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:31:01.949718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:31:01.949718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:31:01.949718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:31:01.949718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:31:01.949718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:31:01.949718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:31:01.949718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:31:01.949718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:31:01.949718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:31:01.949718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:31:01.949718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:31:01.949718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:31:01.949718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:31:01.953514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:31:01.953514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:31:01.953514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:31:01.953514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:31:01.953514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:31:01.953514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:31:01.953514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:31:01.953514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:31:01.953514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:31:01.953514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:31:01.953514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:31:01.953514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:31:01.953514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:31:01.953514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:31:01.953514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:31:01.953514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:31:01.953514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:31:01.954619 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:31:01.954619 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:31:01.954619 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:31:01.954619 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 628/1265] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:31:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:31:01.957320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ceaddd-35f2-4b7c-8c20-ac13ce60c8d6 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:02.039446 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:02.057515 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ceaddd-35f2-4b7c-8c20-ac13ce60c8d6 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:02.090928 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0704acf-aff8-4a21-8d90-9d64a283f20d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:02.091531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0704acf-aff8-4a21-8d90-9d64a283f20d None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:02.091811 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0704acf-aff8-4a21-8d90-9d64a283f20d None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:02.092083 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0704acf-aff8-4a21-8d90-9d64a283f20d None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:02.092715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0704acf-aff8-4a21-8d90-9d64a283f20d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:31:02.093019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0704acf-aff8-4a21-8d90-9d64a283f20d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:02.094394 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0704acf-aff8-4a21-8d90-9d64a283f20d None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:02.105554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 638/1266] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:31:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:31:02.111529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ec05fb-d12b-43e0-872c-1616303571fb None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:02.212198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ec05fb-d12b-43e0-872c-1616303571fb None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:02.231013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:02.245003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e017e3-7e4f-43c8-9007-1710bd74b4e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:02.245623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e017e3-7e4f-43c8-9007-1710bd74b4e3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:02.245902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e017e3-7e4f-43c8-9007-1710bd74b4e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:02.246175 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e017e3-7e4f-43c8-9007-1710bd74b4e3 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:02.254147 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e017e3-7e4f-43c8-9007-1710bd74b4e3 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:02.259018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e017e3-7e4f-43c8-9007-1710bd74b4e3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:02.266513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-f0e017e3-7e4f-43c8-9007-1710bd74b4e3 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 30 09:31:02.266513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:31:02.266513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:31:02.266513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:31:02.266513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:31:02.266513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:31:02.266513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:31:02.266513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:31:02.266513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:31:02.266513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:31:02.266513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:31:02.266513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:31:02.266513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:31:02.266513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:31:02.266513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:31:02.266513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:31:02.267764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:31:02.267764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:31:02.267764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:31:02.267764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:31:02.267764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:31:02.267764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:31:02.267764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:31:02.267764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:31:02.267764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:31:02.267764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:31:02.267764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:31:02.267764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:31:02.267764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:31:02.267764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:31:02.267764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:31:02.267764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:31:02.267764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:31:02.267764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:31:02.268954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:31:02.268954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:31:02.268954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:31:02.268954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:31:02.268954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:31:02.268954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:31:02.268954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:31:02.268954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:31:02.268954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 30 09:31:02.268954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:31:02.268954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 629/1267] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:31:02 2022] GET /identity/v3/users/placement => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:31:02.273451 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0704acf-aff8-4a21-8d90-9d64a283f20d None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:02.370609 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0704acf-aff8-4a21-8d90-9d64a283f20d None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:02.403922 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f5b813-a795-47c6-b805-05c4d112827b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:02.404515 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f5b813-a795-47c6-b805-05c4d112827b None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:02.404793 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f5b813-a795-47c6-b805-05c4d112827b None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:02.405064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f5b813-a795-47c6-b805-05c4d112827b None admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:02.405773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f5b813-a795-47c6-b805-05c4d112827b None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:31:02.406079 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f5b813-a795-47c6-b805-05c4d112827b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:02.407596 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f5b813-a795-47c6-b805-05c4d112827b None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:02.416531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:02.424570 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 639/1268] 10.222.0.39 () {56 vars in 1143 bytes} [Fri Sep 30 09:31:02 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:31:02.431093 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e017e3-7e4f-43c8-9007-1710bd74b4e3 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:02.527926 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e017e3-7e4f-43c8-9007-1710bd74b4e3 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:02.560583 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414e0e9e-c2aa-4e00-aaba-aac4fbfb4cff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:02.561202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414e0e9e-c2aa-4e00-aaba-aac4fbfb4cff None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:02.561481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414e0e9e-c2aa-4e00-aaba-aac4fbfb4cff None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:02.561753 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414e0e9e-c2aa-4e00-aaba-aac4fbfb4cff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:02.572472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414e0e9e-c2aa-4e00-aaba-aac4fbfb4cff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:02.573906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414e0e9e-c2aa-4e00-aaba-aac4fbfb4cff None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:02.587527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-414e0e9e-c2aa-4e00-aaba-aac4fbfb4cff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:31:02.587527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:31:02.587527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:31:02.587527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:31:02.587527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:31:02.587527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:31:02.587527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:31:02.587527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:31:02.587527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:31:02.587527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:31:02.587527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:31:02.587527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:31:02.587527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:31:02.587527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:31:02.587527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:31:02.587527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:31:02.588571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:31:02.588571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:31:02.588571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:31:02.588571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:31:02.588571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:31:02.588571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:31:02.588571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:31:02.588571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:31:02.588571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:31:02.588571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:31:02.588571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:31:02.588571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:31:02.588571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:31:02.588571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:31:02.588571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:31:02.588571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:31:02.588571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:31:02.589601 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:31:02.589601 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:31:02.589601 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:31:02.589601 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 630/1269] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:31:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:31:02.594288 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f5b813-a795-47c6-b805-05c4d112827b None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:02.693824 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f5b813-a795-47c6-b805-05c4d112827b None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:02.727735 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a6bf3a-9d0f-4632-b609-718c27b40a49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:02.728329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a6bf3a-9d0f-4632-b609-718c27b40a49 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:02.728607 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a6bf3a-9d0f-4632-b609-718c27b40a49 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:02.728876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a6bf3a-9d0f-4632-b609-718c27b40a49 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:02.729502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a6bf3a-9d0f-4632-b609-718c27b40a49 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:31:02.729799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a6bf3a-9d0f-4632-b609-718c27b40a49 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:02.731207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a6bf3a-9d0f-4632-b609-718c27b40a49 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:02.742776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 640/1270] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:31:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:31:02.748529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414e0e9e-c2aa-4e00-aaba-aac4fbfb4cff None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:02.778918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:02.846233 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414e0e9e-c2aa-4e00-aaba-aac4fbfb4cff None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:02.881004 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb56d2e4-76a0-4f8e-8961-36498308d078 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:02.881628 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb56d2e4-76a0-4f8e-8961-36498308d078 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:02.881907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb56d2e4-76a0-4f8e-8961-36498308d078 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:02.882180 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb56d2e4-76a0-4f8e-8961-36498308d078 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:02.888555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb56d2e4-76a0-4f8e-8961-36498308d078 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:02.889965 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb56d2e4-76a0-4f8e-8961-36498308d078 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:02.900855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-fb56d2e4-76a0-4f8e-8961-36498308d078 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:31:02.900855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:31:02.900855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:31:02.900855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:31:02.900855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:31:02.900855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:31:02.900855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:31:02.900855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:31:02.900855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:31:02.900855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:31:02.900855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:31:02.900855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:31:02.900855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:31:02.900855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:31:02.900855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:31:02.901830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:31:02.901830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:31:02.901830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:31:02.901830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:31:02.901830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:31:02.901830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:31:02.901830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:31:02.901830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:31:02.901830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:31:02.901830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:31:02.901830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:31:02.901830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:31:02.901830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:31:02.901830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:31:02.901830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:31:02.901830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:31:02.901830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:31:02.903195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:31:02.903195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:31:02.903195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:31:02.903195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:31:02.903195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:31:02.903195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:31:02.903195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:31:02.903195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:31:02.903195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 631/1271] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:31:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 30 09:31:02.907795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a6bf3a-9d0f-4632-b609-718c27b40a49 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:02.966457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:03.007127 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a6bf3a-9d0f-4632-b609-718c27b40a49 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:03.040423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a231ed-0228-4756-a813-fdd1b1a3ec8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:03.041022 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a231ed-0228-4756-a813-fdd1b1a3ec8e None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:03.041298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a231ed-0228-4756-a813-fdd1b1a3ec8e None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:03.041565 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a231ed-0228-4756-a813-fdd1b1a3ec8e None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:03.042241 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a231ed-0228-4756-a813-fdd1b1a3ec8e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:31:03.042540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a231ed-0228-4756-a813-fdd1b1a3ec8e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:03.043997 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a231ed-0228-4756-a813-fdd1b1a3ec8e None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:03.055725 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 641/1272] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:31:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:31:03.061737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb56d2e4-76a0-4f8e-8961-36498308d078 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:03.152160 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:03.160960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb56d2e4-76a0-4f8e-8961-36498308d078 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:03.194352 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df15bc67-a6fc-4a29-a556-a7b1715a859c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:03.195053 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df15bc67-a6fc-4a29-a556-a7b1715a859c None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:03.195327 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df15bc67-a6fc-4a29-a556-a7b1715a859c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:03.195599 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df15bc67-a6fc-4a29-a556-a7b1715a859c None admin] PATH_INFO: `/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/49a3011c103b4e48904f876f81d27fba/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:03.197495 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df15bc67-a6fc-4a29-a556-a7b1715a859c None admin] RBAC: Authorizing `identity:create_grant(project_id=c3802ced1fce4997a016d41eca843ef8, user_id=49a3011c103b4e48904f876f81d27fba, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:03.198985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df15bc67-a6fc-4a29-a556-a7b1715a859c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:03.401269 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 632/1273] 10.222.0.39 () {58 vars in 1439 bytes} [Fri Sep 30 09:31:03 2022] PUT /identity/v3/projects/c3802ced1fce4997a016d41eca843ef8/users/49a3011c103b4e48904f876f81d27fba/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:31:05.541698 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa009616-624a-4e71-86eb-764c978dd14b None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:05.542247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa009616-624a-4e71-86eb-764c978dd14b None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:05.542247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa009616-624a-4e71-86eb-764c978dd14b None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:05.543610 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 642/1274] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:31:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:31:05.548894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665c6898-0cfb-4786-9e12-015451c2d7d4 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:05.549311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665c6898-0cfb-4786-9e12-015451c2d7d4 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:05.549439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665c6898-0cfb-4786-9e12-015451c2d7d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:05.569212 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-665c6898-0cfb-4786-9e12-015451c2d7d4 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:31:05.570427 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-665c6898-0cfb-4786-9e12-015451c2d7d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:31:05.634351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 633/1275] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:31:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:31:05.904861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa009616-624a-4e71-86eb-764c978dd14b None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:06.005151 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa009616-624a-4e71-86eb-764c978dd14b None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:06.022459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:06.038988 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f446b1d2-c6ba-4faf-983a-1235a4eec2ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:06.039618 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f446b1d2-c6ba-4faf-983a-1235a4eec2ec None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:06.039893 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f446b1d2-c6ba-4faf-983a-1235a4eec2ec None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:06.040159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f446b1d2-c6ba-4faf-983a-1235a4eec2ec None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:06.051245 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f446b1d2-c6ba-4faf-983a-1235a4eec2ec None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 30 09:31:06.055976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-f446b1d2-c6ba-4faf-983a-1235a4eec2ec None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:31:06.055976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:31:06.055976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:31:06.055976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:31:06.055976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:31:06.055976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:31:06.055976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:31:06.055976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:31:06.055976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:31:06.055976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:31:06.055976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:31:06.055976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:31:06.055976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 30 09:31:06.055976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 30 09:31:06.055976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 30 09:31:06.055976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 30 09:31:06.058676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 30 09:31:06.058676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 30 09:31:06.058676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 30 09:31:06.058676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 30 09:31:06.058676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:31:06.058676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:31:06.058676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:31:06.058676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:31:06.058676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:31:06.058676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:31:06.058676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:31:06.058676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:31:06.058676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:31:06.058676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:31:06.058676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:31:06.058676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:31:06.059777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:31:06.059777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:31:06.059777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:31:06.059777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:31:06.059777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 30 09:31:06.059777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 30 09:31:06.059777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 30 09:31:06.059777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 30 09:31:06.059777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 30 09:31:06.059777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 30 09:31:06.059777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 30 09:31:06.059777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:31:06.059777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 643/1276] 10.222.0.39 () {56 vars in 1096 bytes} [Fri Sep 30 09:31:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:31:06.063063 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665c6898-0cfb-4786-9e12-015451c2d7d4 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:06.141400 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:06.165265 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665c6898-0cfb-4786-9e12-015451c2d7d4 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:06.193628 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0af9be-1f40-425e-a444-a9a0cd91afd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:06.194761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0af9be-1f40-425e-a444-a9a0cd91afd3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:06.194761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0af9be-1f40-425e-a444-a9a0cd91afd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:06.194954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0af9be-1f40-425e-a444-a9a0cd91afd3 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:06.195430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0af9be-1f40-425e-a444-a9a0cd91afd3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:31:06.195730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0af9be-1f40-425e-a444-a9a0cd91afd3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:06.197092 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0af9be-1f40-425e-a444-a9a0cd91afd3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:06.205069 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 634/1277] 10.222.0.39 () {56 vars in 1099 bytes} [Fri Sep 30 09:31:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:31:06.211376 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f446b1d2-c6ba-4faf-983a-1235a4eec2ec None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:06.309050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f446b1d2-c6ba-4faf-983a-1235a4eec2ec None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:06.341393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0457be6d-937a-4a45-9b0e-5ca5623ded37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:06.341998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0457be6d-937a-4a45-9b0e-5ca5623ded37 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:06.342275 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0457be6d-937a-4a45-9b0e-5ca5623ded37 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:06.342544 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0457be6d-937a-4a45-9b0e-5ca5623ded37 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:06.352866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0457be6d-937a-4a45-9b0e-5ca5623ded37 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:06.360441 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0457be6d-937a-4a45-9b0e-5ca5623ded37 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:06.370029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-0457be6d-937a-4a45-9b0e-5ca5623ded37 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:31:06.370029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:31:06.370029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:31:06.370029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:31:06.370029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:31:06.370029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:31:06.370029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:31:06.370029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:31:06.370029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:31:06.370029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:31:06.370029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:31:06.370029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:31:06.370029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:31:06.370029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:31:06.370029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:31:06.370029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:31:06.371056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:31:06.371056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:31:06.371056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:31:06.371056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:31:06.371056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:31:06.371056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:31:06.371056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:31:06.371056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:31:06.371056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:31:06.371056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:31:06.371056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:31:06.371056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:31:06.371056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:31:06.371056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:31:06.371056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:31:06.371056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:31:06.371056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:31:06.371998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:31:06.371998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:31:06.371998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:31:06.371998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 644/1278] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:31:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:31:06.377120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0af9be-1f40-425e-a444-a9a0cd91afd3 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:06.475745 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0af9be-1f40-425e-a444-a9a0cd91afd3 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:06.508265 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f6fd67-43e1-4cbb-b85f-dedb81da4979 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:06.508862 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f6fd67-43e1-4cbb-b85f-dedb81da4979 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:06.509160 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f6fd67-43e1-4cbb-b85f-dedb81da4979 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:06.509446 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f6fd67-43e1-4cbb-b85f-dedb81da4979 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:06.510078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f6fd67-43e1-4cbb-b85f-dedb81da4979 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:31:06.510377 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f6fd67-43e1-4cbb-b85f-dedb81da4979 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:06.511764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f6fd67-43e1-4cbb-b85f-dedb81da4979 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:06.523840 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 635/1279] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:31:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:31:06.529594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0457be6d-937a-4a45-9b0e-5ca5623ded37 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:06.560370 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:06.628946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0457be6d-937a-4a45-9b0e-5ca5623ded37 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:06.661808 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a8cc98-93db-489a-be7e-0451b5caf9be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:06.662424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a8cc98-93db-489a-be7e-0451b5caf9be None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:06.662730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a8cc98-93db-489a-be7e-0451b5caf9be None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:06.663002 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a8cc98-93db-489a-be7e-0451b5caf9be None admin] PATH_INFO: `/v3/users/placement` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:06.669692 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a8cc98-93db-489a-be7e-0451b5caf9be None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:06.671213 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a8cc98-93db-489a-be7e-0451b5caf9be None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:06.683042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-10a8cc98-93db-489a-be7e-0451b5caf9be None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 30 09:31:06.683042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:31:06.683042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:31:06.683042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:31:06.683042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:31:06.683042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:31:06.683042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:31:06.683042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:31:06.683042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:31:06.683042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:31:06.683042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:31:06.683042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:31:06.683042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 30 09:31:06.683042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 30 09:31:06.683042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 30 09:31:06.683042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 30 09:31:06.687639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:31:06.687639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:31:06.687639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 30 09:31:06.687639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:31:06.687639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 30 09:31:06.687639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 30 09:31:06.687639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:31:06.687639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:31:06.687639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:31:06.687639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:31:06.687639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:31:06.687639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:31:06.687639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:31:06.687639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:31:06.687639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:31:06.687639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:31:06.687639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:31:06.687639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:31:06.688605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:31:06.688605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:31:06.688605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 30 09:31:06.688605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 30 09:31:06.688605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 30 09:31:06.688605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 30 09:31:06.688605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 30 09:31:06.688605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 30 09:31:06.688605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 30 09:31:06.688605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:31:06.688605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 645/1280] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:31:06 2022] GET /identity/v3/users/placement => generated 87 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:31:06.691406 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f6fd67-43e1-4cbb-b85f-dedb81da4979 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:06.695323 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:06.790414 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f6fd67-43e1-4cbb-b85f-dedb81da4979 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:06.827687 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa9b71d-fee9-49e7-a8c2-e05eccdd6c1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:06.828300 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa9b71d-fee9-49e7-a8c2-e05eccdd6c1a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:06.828576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa9b71d-fee9-49e7-a8c2-e05eccdd6c1a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:06.828846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa9b71d-fee9-49e7-a8c2-e05eccdd6c1a None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:06.829576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa9b71d-fee9-49e7-a8c2-e05eccdd6c1a None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:31:06.829876 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa9b71d-fee9-49e7-a8c2-e05eccdd6c1a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:06.831362 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa9b71d-fee9-49e7-a8c2-e05eccdd6c1a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:06.850495 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 636/1281] 10.222.0.39 () {56 vars in 1143 bytes} [Fri Sep 30 09:31:06 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:31:06.856887 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a8cc98-93db-489a-be7e-0451b5caf9be None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:06.954562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:06.964276 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a8cc98-93db-489a-be7e-0451b5caf9be None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:07.000240 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2123fe-f2f9-4437-b609-cb02cb7b06c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:07.000874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2123fe-f2f9-4437-b609-cb02cb7b06c1 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:07.001147 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2123fe-f2f9-4437-b609-cb02cb7b06c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:07.001672 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2123fe-f2f9-4437-b609-cb02cb7b06c1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:07.079189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2123fe-f2f9-4437-b609-cb02cb7b06c1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:07.080579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2123fe-f2f9-4437-b609-cb02cb7b06c1 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:07.172746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:07.183535 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 646/1282] 10.222.0.39 () {58 vars in 1282 bytes} [Fri Sep 30 09:31:06 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 30 09:31:07.189705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa9b71d-fee9-49e7-a8c2-e05eccdd6c1a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:07.253127 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:07.290609 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa9b71d-fee9-49e7-a8c2-e05eccdd6c1a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:07.324832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2293d8ff-515d-4b94-b40c-424b534b49dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:07.325545 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2293d8ff-515d-4b94-b40c-424b534b49dc None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:07.325827 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2293d8ff-515d-4b94-b40c-424b534b49dc None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:07.326099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2293d8ff-515d-4b94-b40c-424b534b49dc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:07.339273 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2293d8ff-515d-4b94-b40c-424b534b49dc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:07.339273 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2293d8ff-515d-4b94-b40c-424b534b49dc None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:07.352867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-2293d8ff-515d-4b94-b40c-424b534b49dc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:31:07.352867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:31:07.352867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:31:07.352867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:31:07.352867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:31:07.352867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:31:07.352867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:31:07.352867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:31:07.352867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:31:07.352867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:31:07.352867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:31:07.352867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:31:07.352867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 30 09:31:07.352867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 30 09:31:07.352867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 30 09:31:07.352867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 30 09:31:07.353831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:31:07.353831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:31:07.353831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:31:07.353831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:31:07.353831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:31:07.353831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:31:07.353831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:31:07.353831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:31:07.353831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:31:07.353831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:31:07.353831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:31:07.353831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:31:07.353831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:31:07.353831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:31:07.353831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:31:07.353831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:31:07.353831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 30 09:31:07.354806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 30 09:31:07.354806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 30 09:31:07.354806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:31:07.354806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 637/1283] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:31:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:31:07.359963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2123fe-f2f9-4437-b609-cb02cb7b06c1 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:07.455811 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2123fe-f2f9-4437-b609-cb02cb7b06c1 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:07.489358 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4071cb-add0-471c-8fb7-86df8fa518b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:07.489951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4071cb-add0-471c-8fb7-86df8fa518b9 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:07.490227 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4071cb-add0-471c-8fb7-86df8fa518b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:07.490496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4071cb-add0-471c-8fb7-86df8fa518b9 None admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:07.491155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4071cb-add0-471c-8fb7-86df8fa518b9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:31:07.491457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4071cb-add0-471c-8fb7-86df8fa518b9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:07.492821 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4071cb-add0-471c-8fb7-86df8fa518b9 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:07.504062 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 647/1284] 10.222.0.39 () {56 vars in 1109 bytes} [Fri Sep 30 09:31:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:31:07.509645 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2293d8ff-515d-4b94-b40c-424b534b49dc None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:07.608836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2293d8ff-515d-4b94-b40c-424b534b49dc None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:07.621836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:07.640983 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be553f1-5a2b-472d-8356-ba737171a0fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:07.641594 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be553f1-5a2b-472d-8356-ba737171a0fb None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:07.641876 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be553f1-5a2b-472d-8356-ba737171a0fb None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:07.642152 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be553f1-5a2b-472d-8356-ba737171a0fb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:07.649131 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be553f1-5a2b-472d-8356-ba737171a0fb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:07.650546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be553f1-5a2b-472d-8356-ba737171a0fb None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:07.662037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-4be553f1-5a2b-472d-8356-ba737171a0fb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:31:07.662037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:31:07.662037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:31:07.662037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:31:07.662037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:31:07.662037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:31:07.662037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:31:07.662037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:31:07.662037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:31:07.662037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:31:07.662037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:31:07.662037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:31:07.662037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 30 09:31:07.662037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 30 09:31:07.662037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 30 09:31:07.663128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 30 09:31:07.663128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:31:07.663128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:31:07.663128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:31:07.663128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:31:07.663128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:31:07.663128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:31:07.663128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:31:07.663128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 30 09:31:07.663128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:31:07.663128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:31:07.663128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:31:07.663128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:31:07.663128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:31:07.663128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:31:07.663128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:31:07.663128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:31:07.664356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 30 09:31:07.664356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 30 09:31:07.664356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 30 09:31:07.664356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 30 09:31:07.664356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 30 09:31:07.664356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 30 09:31:07.664356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 30 09:31:07.664356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 30 09:31:07.664356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 638/1285] 10.222.0.39 () {56 vars in 1111 bytes} [Fri Sep 30 09:31:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:31:07.669009 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4071cb-add0-471c-8fb7-86df8fa518b9 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:07.718067 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:07.766971 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4071cb-add0-471c-8fb7-86df8fa518b9 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:07.800648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ba5f23-81e3-47f8-9415-4bd41a589388 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:07.801242 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ba5f23-81e3-47f8-9415-4bd41a589388 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:07.801522 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ba5f23-81e3-47f8-9415-4bd41a589388 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:07.801792 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ba5f23-81e3-47f8-9415-4bd41a589388 None admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:07.802467 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ba5f23-81e3-47f8-9415-4bd41a589388 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:31:07.802799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ba5f23-81e3-47f8-9415-4bd41a589388 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:07.804231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ba5f23-81e3-47f8-9415-4bd41a589388 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:07.815653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 648/1286] 10.222.0.39 () {56 vars in 1148 bytes} [Fri Sep 30 09:31:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:31:07.821793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be553f1-5a2b-472d-8356-ba737171a0fb None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:07.918847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be553f1-5a2b-472d-8356-ba737171a0fb None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:07.950497 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4dc2f8-8cff-4c12-8adc-fc31aa93994e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:07.951148 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4dc2f8-8cff-4c12-8adc-fc31aa93994e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:07.951426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4dc2f8-8cff-4c12-8adc-fc31aa93994e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:07.951702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4dc2f8-8cff-4c12-8adc-fc31aa93994e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:07.952402 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4dc2f8-8cff-4c12-8adc-fc31aa93994e None admin] RBAC: Adding query filter params (role.id=e874c311084c4c9fb7691bce461d319b, scope.project.id=c3802ced1fce4997a016d41eca843ef8, user.id=49a3011c103b4e48904f876f81d27fba) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:31:07.952707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4dc2f8-8cff-4c12-8adc-fc31aa93994e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:07.954122 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4dc2f8-8cff-4c12-8adc-fc31aa93994e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:07.960484 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 639/1287] 10.222.0.39 () {56 vars in 1374 bytes} [Fri Sep 30 09:31:07 2022] GET /identity/v3/role_assignments?user.id=49a3011c103b4e48904f876f81d27fba&scope.project.id=c3802ced1fce4997a016d41eca843ef8&role.id=e874c311084c4c9fb7691bce461d319b => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:31:10.128878 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966caab5-adb4-4b89-823a-0288b9f9312b None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:10.129436 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966caab5-adb4-4b89-823a-0288b9f9312b None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:10.129436 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966caab5-adb4-4b89-823a-0288b9f9312b None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:10.130799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 649/1288] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:31:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:31:10.135988 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2a75fa-646c-4308-b60f-f6b78394acac None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:10.136402 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2a75fa-646c-4308-b60f-f6b78394acac None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:10.136536 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2a75fa-646c-4308-b60f-f6b78394acac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:10.154818 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-bb2a75fa-646c-4308-b60f-f6b78394acac None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:31:10.156019 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-bb2a75fa-646c-4308-b60f-f6b78394acac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:31:10.185610 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:10.222732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 640/1289] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:31:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:31:10.492845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966caab5-adb4-4b89-823a-0288b9f9312b None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:10.592476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966caab5-adb4-4b89-823a-0288b9f9312b None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:10.603440 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:10.626358 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa823339-7584-4224-bac6-7f212cbfbedf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:10.626993 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa823339-7584-4224-bac6-7f212cbfbedf None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:10.627268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa823339-7584-4224-bac6-7f212cbfbedf None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:10.627533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa823339-7584-4224-bac6-7f212cbfbedf None admin] PATH_INFO: `/v3/services/placement` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:10.628139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa823339-7584-4224-bac6-7f212cbfbedf None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:10.629534 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa823339-7584-4224-bac6-7f212cbfbedf None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:10.638069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-aa823339-7584-4224-bac6-7f212cbfbedf None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 30 09:31:10.638069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:31:10.638069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:31:10.638069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:31:10.638069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:31:10.638069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:31:10.638069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:31:10.638069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:31:10.638069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 30 09:31:10.638069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:31:10.638069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:31:10.638069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:31:10.638069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:31:10.638069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:31:10.638069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:31:10.638069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:31:10.638069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:31:10.644117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:31:10.644117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:31:10.644117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:31:10.644117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:31:10.644117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:31:10.644117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:31:10.644117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:31:10.644117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:31:10.644117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:31:10.644117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:31:10.644117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:31:10.644117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:31:10.644117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:31:10.644117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:31:10.644117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:31:10.644117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:31:10.645096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:31:10.645096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:31:10.645096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:31:10.645096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:31:10.645096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:31:10.645096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:31:10.645096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:31:10.645096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:31:10.645096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:31:10.645096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:31:10.645096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:31:10.645096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:31:10.645096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 30 09:31:10.645096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:31:10.645096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 650/1290] 10.222.0.39 () {56 vars in 1117 bytes} [Fri Sep 30 09:31:10 2022] GET /identity/v3/services/placement => generated 90 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:31:10.647325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2a75fa-646c-4308-b60f-f6b78394acac None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:10.740228 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2a75fa-646c-4308-b60f-f6b78394acac None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:10.772904 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b65664-18c5-4166-b080-5b5be13daa19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:10.773444 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b65664-18c5-4166-b080-5b5be13daa19 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:10.773718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b65664-18c5-4166-b080-5b5be13daa19 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:10.773983 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b65664-18c5-4166-b080-5b5be13daa19 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:10.774583 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b65664-18c5-4166-b080-5b5be13daa19 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:31:10.774905 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b65664-18c5-4166-b080-5b5be13daa19 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:10.776358 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b65664-18c5-4166-b080-5b5be13daa19 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:10.780985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 641/1291] 10.222.0.39 () {56 vars in 1116 bytes} [Fri Sep 30 09:31:10 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:31:10.787535 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa823339-7584-4224-bac6-7f212cbfbedf None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:10.885201 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa823339-7584-4224-bac6-7f212cbfbedf None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:10.917206 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367f3bab-05ea-4e60-8790-aed44a691380 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:10.917794 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367f3bab-05ea-4e60-8790-aed44a691380 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:10.918071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367f3bab-05ea-4e60-8790-aed44a691380 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:10.918340 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367f3bab-05ea-4e60-8790-aed44a691380 None admin] PATH_INFO: `/v3/services` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:10.918983 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367f3bab-05ea-4e60-8790-aed44a691380 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:31:10.919287 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367f3bab-05ea-4e60-8790-aed44a691380 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:10.920661 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367f3bab-05ea-4e60-8790-aed44a691380 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:10.924940 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 651/1292] 10.222.0.39 () {56 vars in 1116 bytes} [Fri Sep 30 09:31:10 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:31:13.034505 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3936bcb0-5b89-4b0f-9be7-e1ecfdfb72d6 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:13.035190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3936bcb0-5b89-4b0f-9be7-e1ecfdfb72d6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:13.035190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3936bcb0-5b89-4b0f-9be7-e1ecfdfb72d6 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:13.037007 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 642/1293] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:31:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:31:13.042100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f98c12-1e35-4757-bc7c-a1019c04612a None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:13.042542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f98c12-1e35-4757-bc7c-a1019c04612a None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:13.042672 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f98c12-1e35-4757-bc7c-a1019c04612a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:13.060603 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-53f98c12-1e35-4757-bc7c-a1019c04612a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:31:13.067025 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-53f98c12-1e35-4757-bc7c-a1019c04612a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:31:13.090345 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:13.126337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 652/1294] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:31:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:31:13.392054 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3936bcb0-5b89-4b0f-9be7-e1ecfdfb72d6 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:13.422388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:13.490532 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3936bcb0-5b89-4b0f-9be7-e1ecfdfb72d6 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:13.524536 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e83862d-4b56-4084-90a6-4ae72a49baa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:13.525128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e83862d-4b56-4084-90a6-4ae72a49baa8 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:13.525402 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e83862d-4b56-4084-90a6-4ae72a49baa8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:13.525670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e83862d-4b56-4084-90a6-4ae72a49baa8 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:13.526301 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e83862d-4b56-4084-90a6-4ae72a49baa8 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:13.527683 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e83862d-4b56-4084-90a6-4ae72a49baa8 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:13.599714 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 643/1295] 10.222.0.39 () {60 vars in 1141 bytes} [Fri Sep 30 09:31:13 2022] POST /identity/v3/services => generated 246 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:31:15.744518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83153ae-6483-4f3d-a492-5364ed53c4db None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:15.745070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83153ae-6483-4f3d-a492-5364ed53c4db None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:15.745070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83153ae-6483-4f3d-a492-5364ed53c4db None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:15.746425 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 653/1296] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:31:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:31:15.751924 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fadb326-8239-4cf6-8476-9725abec53c8 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:15.758149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fadb326-8239-4cf6-8476-9725abec53c8 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:15.758149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fadb326-8239-4cf6-8476-9725abec53c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:15.771311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-2fadb326-8239-4cf6-8476-9725abec53c8 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:31:15.772477 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2fadb326-8239-4cf6-8476-9725abec53c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:31:15.841602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 644/1297] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:31:15 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:31:16.106562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e83153ae-6483-4f3d-a492-5364ed53c4db None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:16.206980 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e83153ae-6483-4f3d-a492-5364ed53c4db None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:16.240863 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25538c5-5101-4e52-a72a-f5ee0b91b084 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:16.241480 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25538c5-5101-4e52-a72a-f5ee0b91b084 None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:16.241758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25538c5-5101-4e52-a72a-f5ee0b91b084 None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:16.242050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25538c5-5101-4e52-a72a-f5ee0b91b084 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:16.242635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25538c5-5101-4e52-a72a-f5ee0b91b084 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:16.244027 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25538c5-5101-4e52-a72a-f5ee0b91b084 None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:16.252838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-b25538c5-5101-4e52-a72a-f5ee0b91b084 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 30 09:31:16.252838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:31:16.252838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:31:16.252838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:31:16.252838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:31:16.252838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:31:16.252838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:31:16.252838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:31:16.252838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 30 09:31:16.252838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:31:16.252838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:31:16.252838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:31:16.252838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:31:16.252838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:31:16.252838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:31:16.252838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:31:16.252838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:31:16.254188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:31:16.254188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:31:16.254188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:31:16.254188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:31:16.254188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:31:16.254188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:31:16.254188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:31:16.254188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:31:16.254188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:31:16.254188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:31:16.254188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:31:16.254188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:31:16.254188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:31:16.254188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:31:16.254188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:31:16.254188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:31:16.255360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:31:16.255360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:31:16.255360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:31:16.255360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:31:16.255360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:31:16.255360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:31:16.255360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:31:16.255360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:31:16.255360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:31:16.255360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:31:16.255360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:31:16.255360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:31:16.255360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 30 09:31:16.255360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:31:16.255360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 654/1298] 10.222.0.39 () {56 vars in 1117 bytes} [Fri Sep 30 09:31:16 2022] GET /identity/v3/services/placement => generated 90 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:31:16.259964 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fadb326-8239-4cf6-8476-9725abec53c8 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:16.277397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:16.357943 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fadb326-8239-4cf6-8476-9725abec53c8 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:16.391361 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3954cd9-cb4b-4bbe-8f10-5bcb2daa9cb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:16.391956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3954cd9-cb4b-4bbe-8f10-5bcb2daa9cb5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:16.392230 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3954cd9-cb4b-4bbe-8f10-5bcb2daa9cb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:16.392495 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3954cd9-cb4b-4bbe-8f10-5bcb2daa9cb5 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:16.393098 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3954cd9-cb4b-4bbe-8f10-5bcb2daa9cb5 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:31:16.393396 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3954cd9-cb4b-4bbe-8f10-5bcb2daa9cb5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:16.394773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3954cd9-cb4b-4bbe-8f10-5bcb2daa9cb5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:16.399687 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 645/1299] 10.222.0.39 () {56 vars in 1116 bytes} [Fri Sep 30 09:31:16 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:31:16.405743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25538c5-5101-4e52-a72a-f5ee0b91b084 None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:16.437474 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:16.503977 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25538c5-5101-4e52-a72a-f5ee0b91b084 None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:16.538142 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d07597-3f8b-475e-8fbe-a135fa44e24e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:16.538762 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d07597-3f8b-475e-8fbe-a135fa44e24e None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:16.539041 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d07597-3f8b-475e-8fbe-a135fa44e24e None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:16.539310 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d07597-3f8b-475e-8fbe-a135fa44e24e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:16.539994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d07597-3f8b-475e-8fbe-a135fa44e24e None admin] RBAC: Adding query filter params (interface=public, service_id=addf304f0f6a43fa80907db9215becbc, region_id=RegionOne) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:31:16.540297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d07597-3f8b-475e-8fbe-a135fa44e24e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:16.541760 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d07597-3f8b-475e-8fbe-a135fa44e24e None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:16.546694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 655/1300] 10.222.0.39 () {56 vars in 1251 bytes} [Fri Sep 30 09:31:16 2022] GET /identity/v3/endpoints?service_id=addf304f0f6a43fa80907db9215becbc&interface=public®ion_id=RegionOne => generated 194 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:31:16.552297 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3954cd9-cb4b-4bbe-8f10-5bcb2daa9cb5 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:16.629718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:16.646928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3954cd9-cb4b-4bbe-8f10-5bcb2daa9cb5 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:16.679613 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751f812c-414b-4b9e-918f-9936e6137394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:16.680221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751f812c-414b-4b9e-918f-9936e6137394 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:16.680502 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751f812c-414b-4b9e-918f-9936e6137394 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:16.680778 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751f812c-414b-4b9e-918f-9936e6137394 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:16.681369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751f812c-414b-4b9e-918f-9936e6137394 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:16.682763 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751f812c-414b-4b9e-918f-9936e6137394 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:16.689407 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 646/1301] 10.222.0.39 () {56 vars in 1087 bytes} [Fri Sep 30 09:31:16 2022] GET /identity/v3/services => generated 2191 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:31:18.825220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be0cd43-512a-4cf8-b04b-6c89aacd2798 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:18.825797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be0cd43-512a-4cf8-b04b-6c89aacd2798 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:18.825797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be0cd43-512a-4cf8-b04b-6c89aacd2798 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:18.827124 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 656/1302] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:31:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:31:18.832228 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11f2f86-1a90-496f-aa2c-e7f7b884bf96 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:18.832660 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11f2f86-1a90-496f-aa2c-e7f7b884bf96 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:18.832760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11f2f86-1a90-496f-aa2c-e7f7b884bf96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:18.850774 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c11f2f86-1a90-496f-aa2c-e7f7b884bf96 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:31:18.851940 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c11f2f86-1a90-496f-aa2c-e7f7b884bf96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:31:18.916547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 647/1303] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:31:18 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 30 09:31:19.184606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be0cd43-512a-4cf8-b04b-6c89aacd2798 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:19.276319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:19.282780 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be0cd43-512a-4cf8-b04b-6c89aacd2798 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:19.316384 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0cab3e-04ce-48db-b45f-3d15f57973bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:19.317007 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0cab3e-04ce-48db-b45f-3d15f57973bb None admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:19.317284 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0cab3e-04ce-48db-b45f-3d15f57973bb None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:19.317552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0cab3e-04ce-48db-b45f-3d15f57973bb None admin] PATH_INFO: `/v3/services/placement` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:19.318117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0cab3e-04ce-48db-b45f-3d15f57973bb None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:19.319517 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0cab3e-04ce-48db-b45f-3d15f57973bb None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:19.328192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application [None req-fc0cab3e-04ce-48db-b45f-3d15f57973bb None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 30 09:31:19.328192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:31:19.328192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 30 09:31:19.328192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 30 09:31:19.328192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 30 09:31:19.328192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 30 09:31:19.328192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 30 09:31:19.328192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:31:19.328192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 30 09:31:19.328192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:31:19.328192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 30 09:31:19.328192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:31:19.328192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 30 09:31:19.328192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 30 09:31:19.328192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 30 09:31:19.328192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 30 09:31:19.328192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 30 09:31:19.333019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 30 09:31:19.333019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 30 09:31:19.333019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 30 09:31:19.333019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 30 09:31:19.333019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 30 09:31:19.333019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 30 09:31:19.333019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 30 09:31:19.333019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 30 09:31:19.333019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 30 09:31:19.333019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 30 09:31:19.333019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 30 09:31:19.333019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 30 09:31:19.333019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 30 09:31:19.333019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 30 09:31:19.333019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 30 09:31:19.333019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.get_or_create( Sep 30 09:31:19.334078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 30 09:31:19.334078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application with Lock( Sep 30 09:31:19.334078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 30 09:31:19.334078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self._enter() Sep 30 09:31:19.334078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 30 09:31:19.334078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 30 09:31:19.334078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 30 09:31:19.334078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application return self.creator() Sep 30 09:31:19.334078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 30 09:31:19.334078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application created_value = creator( Sep 30 09:31:19.334078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 30 09:31:19.334078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 30 09:31:19.334078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 30 09:31:19.334078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: ERROR keystone.server.flask.application Sep 30 09:31:19.334078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 657/1304] 10.222.0.39 () {56 vars in 1117 bytes} [Fri Sep 30 09:31:19 2022] GET /identity/v3/services/placement => generated 90 bytes in 145 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 30 09:31:19.335396 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11f2f86-1a90-496f-aa2c-e7f7b884bf96 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:19.410061 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:19.432319 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11f2f86-1a90-496f-aa2c-e7f7b884bf96 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:19.464543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7f3e12-e17c-41c0-a775-dd9670669337 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:19.465237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7f3e12-e17c-41c0-a775-dd9670669337 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:19.465569 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7f3e12-e17c-41c0-a775-dd9670669337 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:19.466226 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7f3e12-e17c-41c0-a775-dd9670669337 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:19.466880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7f3e12-e17c-41c0-a775-dd9670669337 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:31:19.467188 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7f3e12-e17c-41c0-a775-dd9670669337 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:19.468568 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7f3e12-e17c-41c0-a775-dd9670669337 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:19.474078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 648/1305] 10.222.0.39 () {56 vars in 1116 bytes} [Fri Sep 30 09:31:19 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:31:19.479905 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0cab3e-04ce-48db-b45f-3d15f57973bb None admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:31:19.577474 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0cab3e-04ce-48db-b45f-3d15f57973bb None admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:31:19.610331 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e305a477-1f97-4ae9-858e-27c0e31fb98c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:31:19.610948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e305a477-1f97-4ae9-858e-27c0e31fb98c None admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:31:19.611253 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e305a477-1f97-4ae9-858e-27c0e31fb98c None admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:31:19.611527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e305a477-1f97-4ae9-858e-27c0e31fb98c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:31:19.612176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e305a477-1f97-4ae9-858e-27c0e31fb98c None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:31:19.613578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e305a477-1f97-4ae9-858e-27c0e31fb98c None admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:31:19.701798 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 658/1306] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:31:19 2022] POST /identity/v3/endpoints => generated 331 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:31:29.471079 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=30 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:31:29.697373 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:37:24.862743 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7a9636-1145-406e-b7d9-6e8ce1c277c1 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:24.863317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7a9636-1145-406e-b7d9-6e8ce1c277c1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:24.863317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7a9636-1145-406e-b7d9-6e8ce1c277c1 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:24.867010 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 649/1307] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:37:24 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:37:24.873299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0787a1d3-5807-4182-af0f-38c8caa4815a None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:24.873781 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0787a1d3-5807-4182-af0f-38c8caa4815a None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:24.873864 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0787a1d3-5807-4182-af0f-38c8caa4815a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:24.962458 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-0787a1d3-5807-4182-af0f-38c8caa4815a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:37:24.963862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0787a1d3-5807-4182-af0f-38c8caa4815a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:25.086636 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 659/1308] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:37:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:37:25.385697 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a100ecdf-1e09-4323-8154-57cfac7a78e4 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:25.385987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a100ecdf-1e09-4323-8154-57cfac7a78e4 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:25.386427 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a100ecdf-1e09-4323-8154-57cfac7a78e4 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:25.387882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 650/1309] 10.222.0.39 () {54 vars in 964 bytes} [Fri Sep 30 09:37:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:37:25.394567 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf5d455-c111-4a21-a54d-2dcf5ee58249 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:25.394904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf5d455-c111-4a21-a54d-2dcf5ee58249 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:25.395153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf5d455-c111-4a21-a54d-2dcf5ee58249 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:25.477305 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-2bf5d455-c111-4a21-a54d-2dcf5ee58249 None None] MFA Rules not processed for user `9266babd21414ae9998716e5a0478e9d`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:37:25.478507 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-2bf5d455-c111-4a21-a54d-2dcf5ee58249 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:25.500295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:37:25.557559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 660/1310] 10.222.0.39 () {58 vars in 1061 bytes} [Fri Sep 30 09:37:25 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:37:25.566647 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a100ecdf-1e09-4323-8154-57cfac7a78e4 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:37:25.568619 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a100ecdf-1e09-4323-8154-57cfac7a78e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:25.690631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a100ecdf-1e09-4323-8154-57cfac7a78e4 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:37:25.777232 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e8b41f-6fd0-4f52-9f60-6052a1a27097 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:37:25.778245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e8b41f-6fd0-4f52-9f60-6052a1a27097 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:25.778658 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e8b41f-6fd0-4f52-9f60-6052a1a27097 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:25.779145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e8b41f-6fd0-4f52-9f60-6052a1a27097 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:25.781721 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-98e8b41f-6fd0-4f52-9f60-6052a1a27097 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:25.843900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:37:25.908224 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e8b41f-6fd0-4f52-9f60-6052a1a27097 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:37:25.910544 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e8b41f-6fd0-4f52-9f60-6052a1a27097 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:37:25.934202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 651/1311] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:37:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:37:28.415498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1160428-8c35-47a4-9397-0a4684422f52 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:28.416003 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1160428-8c35-47a4-9397-0a4684422f52 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:28.416003 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1160428-8c35-47a4-9397-0a4684422f52 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:28.417358 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 661/1312] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:37:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:37:28.422651 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd200b6-487b-4eed-9c81-701eef33f09a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:28.423079 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd200b6-487b-4eed-9c81-701eef33f09a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:28.423176 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd200b6-487b-4eed-9c81-701eef33f09a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:28.501805 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-bfd200b6-487b-4eed-9c81-701eef33f09a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:37:28.503031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-bfd200b6-487b-4eed-9c81-701eef33f09a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:28.599264 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 652/1313] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:37:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:37:28.877999 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1160428-8c35-47a4-9397-0a4684422f52 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:37:28.879602 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d1160428-8c35-47a4-9397-0a4684422f52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:28.981474 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1160428-8c35-47a4-9397-0a4684422f52 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:37:29.051390 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:37:29.053703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9ffc00-ba74-4232-bf12-fc628566f390 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:37:29.054294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9ffc00-ba74-4232-bf12-fc628566f390 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:29.054571 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9ffc00-ba74-4232-bf12-fc628566f390 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:29.054869 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9ffc00-ba74-4232-bf12-fc628566f390 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:29.056446 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-eb9ffc00-ba74-4232-bf12-fc628566f390 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:29.161376 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9ffc00-ba74-4232-bf12-fc628566f390 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:37:29.163366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9ffc00-ba74-4232-bf12-fc628566f390 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:37:29.181276 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 662/1314] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:37:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:37:31.864411 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a518e3-c90c-462f-af44-9d2d3cbab0dc None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:31.864948 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a518e3-c90c-462f-af44-9d2d3cbab0dc None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:31.864948 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a518e3-c90c-462f-af44-9d2d3cbab0dc None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:31.866351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 653/1315] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:37:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:37:31.871532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebd9ddb-4c96-4246-942a-2d2315a14fe6 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:31.871964 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebd9ddb-4c96-4246-942a-2d2315a14fe6 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:31.872112 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebd9ddb-4c96-4246-942a-2d2315a14fe6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:31.937782 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-7ebd9ddb-4c96-4246-942a-2d2315a14fe6 None None] MFA Rules not processed for user `dd33c1df1ffc427888650d2537b8f6d7`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:37:31.939025 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7ebd9ddb-4c96-4246-942a-2d2315a14fe6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:31.989692 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:37:32.034310 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 663/1316] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:37:31 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:37:32.319266 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a518e3-c90c-462f-af44-9d2d3cbab0dc None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:37:32.320851 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-86a518e3-c90c-462f-af44-9d2d3cbab0dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:32.330384 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:37:32.421587 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a518e3-c90c-462f-af44-9d2d3cbab0dc None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:37:32.492616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea78aa2e-7df2-4ea3-9a3f-ae1533e0db1c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:37:32.493226 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea78aa2e-7df2-4ea3-9a3f-ae1533e0db1c service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:32.493502 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea78aa2e-7df2-4ea3-9a3f-ae1533e0db1c service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:32.493773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea78aa2e-7df2-4ea3-9a3f-ae1533e0db1c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:32.495379 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ea78aa2e-7df2-4ea3-9a3f-ae1533e0db1c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:32.556039 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:37:32.601369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea78aa2e-7df2-4ea3-9a3f-ae1533e0db1c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:37:32.602825 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea78aa2e-7df2-4ea3-9a3f-ae1533e0db1c service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:37:32.620494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 654/1317] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:37:32 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:37:35.512125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb8c03a-1df6-4fe0-b995-a687e052027b None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:35.512679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb8c03a-1df6-4fe0-b995-a687e052027b None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:35.512679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb8c03a-1df6-4fe0-b995-a687e052027b None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:35.514025 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 664/1318] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:37:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:37:35.519278 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385f5262-cfb3-4c14-a748-2c5b140aa6c8 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:35.519704 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385f5262-cfb3-4c14-a748-2c5b140aa6c8 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:35.519828 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385f5262-cfb3-4c14-a748-2c5b140aa6c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:35.585838 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-385f5262-cfb3-4c14-a748-2c5b140aa6c8 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:37:35.587095 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-385f5262-cfb3-4c14-a748-2c5b140aa6c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:35.688537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 655/1319] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:37:35 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:37:35.971808 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb8c03a-1df6-4fe0-b995-a687e052027b None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:37:35.973360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-2bb8c03a-1df6-4fe0-b995-a687e052027b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:36.074987 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb8c03a-1df6-4fe0-b995-a687e052027b None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:37:36.147141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad052898-6e4f-480f-90bb-a8b10bd98d0e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:37:36.147735 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad052898-6e4f-480f-90bb-a8b10bd98d0e service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:36.148014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad052898-6e4f-480f-90bb-a8b10bd98d0e service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:36.148284 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad052898-6e4f-480f-90bb-a8b10bd98d0e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:36.149874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ad052898-6e4f-480f-90bb-a8b10bd98d0e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:36.156706 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:37:36.258306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad052898-6e4f-480f-90bb-a8b10bd98d0e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:37:36.259773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad052898-6e4f-480f-90bb-a8b10bd98d0e service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:37:36.276955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 665/1320] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:37:35 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:37:39.322858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50aa1f1-9c25-4a45-b153-3a9e86edbd71 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:39.323388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50aa1f1-9c25-4a45-b153-3a9e86edbd71 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:39.323561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50aa1f1-9c25-4a45-b153-3a9e86edbd71 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:39.324879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 656/1321] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:37:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:37:39.330346 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede34d7b-0410-43ba-a409-614e1afc2141 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:39.330790 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede34d7b-0410-43ba-a409-614e1afc2141 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:39.330932 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede34d7b-0410-43ba-a409-614e1afc2141 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:39.396776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-ede34d7b-0410-43ba-a409-614e1afc2141 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:37:39.397967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ede34d7b-0410-43ba-a409-614e1afc2141 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:39.455502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:37:39.498229 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 666/1322] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:37:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:37:39.780110 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50aa1f1-9c25-4a45-b153-3a9e86edbd71 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:37:39.781673 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a50aa1f1-9c25-4a45-b153-3a9e86edbd71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:39.801755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:37:39.888114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50aa1f1-9c25-4a45-b153-3a9e86edbd71 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:37:39.962511 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516ee25e-723a-4314-be17-589d0bf20136 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:37:39.963152 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516ee25e-723a-4314-be17-589d0bf20136 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:39.963435 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516ee25e-723a-4314-be17-589d0bf20136 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:39.963708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516ee25e-723a-4314-be17-589d0bf20136 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:39.965302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-516ee25e-723a-4314-be17-589d0bf20136 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:40.039056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:37:40.076635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516ee25e-723a-4314-be17-589d0bf20136 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:37:40.078452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516ee25e-723a-4314-be17-589d0bf20136 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:37:40.097260 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 657/1323] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:37:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:37:42.304651 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd19a8f-918a-4fff-a5d5-7e8fc4edd2b9 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:42.305202 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd19a8f-918a-4fff-a5d5-7e8fc4edd2b9 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:42.305202 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd19a8f-918a-4fff-a5d5-7e8fc4edd2b9 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:42.306608 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 667/1324] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:37:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:37:42.312537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052ae4af-dfbd-4bde-aadb-662c3e0bba22 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:42.313081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052ae4af-dfbd-4bde-aadb-662c3e0bba22 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:42.313081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052ae4af-dfbd-4bde-aadb-662c3e0bba22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:42.380578 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-052ae4af-dfbd-4bde-aadb-662c3e0bba22 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:37:42.382073 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-052ae4af-dfbd-4bde-aadb-662c3e0bba22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:42.477769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 658/1325] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:37:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:37:42.755805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd19a8f-918a-4fff-a5d5-7e8fc4edd2b9 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:37:42.757603 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-bcd19a8f-918a-4fff-a5d5-7e8fc4edd2b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:42.861536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd19a8f-918a-4fff-a5d5-7e8fc4edd2b9 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:37:42.933880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd22999-bb4d-400a-96d3-1863bb44047d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:37:42.934483 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd22999-bb4d-400a-96d3-1863bb44047d service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:42.934787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd22999-bb4d-400a-96d3-1863bb44047d service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:42.935052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd22999-bb4d-400a-96d3-1863bb44047d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:42.936624 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-4bd22999-bb4d-400a-96d3-1863bb44047d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:42.954216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:37:43.045051 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd22999-bb4d-400a-96d3-1863bb44047d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:37:43.046404 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd22999-bb4d-400a-96d3-1863bb44047d service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:37:43.064693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 668/1326] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:37:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:37:45.285731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3b9812-c3ae-461c-b9c9-c02d9bc8333b None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:45.286248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3b9812-c3ae-461c-b9c9-c02d9bc8333b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:45.286341 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3b9812-c3ae-461c-b9c9-c02d9bc8333b None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:45.288315 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 659/1327] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:37:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:37:45.293471 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24296669-65a5-472b-8bfd-bdbc2a2d193a None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:45.293920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24296669-65a5-472b-8bfd-bdbc2a2d193a None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:45.294023 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24296669-65a5-472b-8bfd-bdbc2a2d193a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:45.360061 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-24296669-65a5-472b-8bfd-bdbc2a2d193a None None] MFA Rules not processed for user `dd33c1df1ffc427888650d2537b8f6d7`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:37:45.361207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-24296669-65a5-472b-8bfd-bdbc2a2d193a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:45.426888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:37:45.459751 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 669/1328] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:37:45 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:37:45.748005 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3b9812-c3ae-461c-b9c9-c02d9bc8333b None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:37:45.749558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fb3b9812-c3ae-461c-b9c9-c02d9bc8333b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:45.783896 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:37:45.860605 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3b9812-c3ae-461c-b9c9-c02d9bc8333b None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:37:45.935907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da7a022-f7e7-4a67-aae8-07fef4025d0d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:37:45.936513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da7a022-f7e7-4a67-aae8-07fef4025d0d service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:45.936787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da7a022-f7e7-4a67-aae8-07fef4025d0d service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:45.937053 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da7a022-f7e7-4a67-aae8-07fef4025d0d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:45.938639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0da7a022-f7e7-4a67-aae8-07fef4025d0d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:46.016218 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:37:46.047464 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da7a022-f7e7-4a67-aae8-07fef4025d0d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:37:46.049458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da7a022-f7e7-4a67-aae8-07fef4025d0d service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:37:46.067706 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 660/1329] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:37:45 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:37:48.611785 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af64128b-1ce5-4f43-9f28-c0dd1f21a97d None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:48.612325 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af64128b-1ce5-4f43-9f28-c0dd1f21a97d None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:48.612325 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af64128b-1ce5-4f43-9f28-c0dd1f21a97d None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:48.613688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 670/1330] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:37:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:37:48.619128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b401ae5-f38b-4ba5-ac39-2a4b4ca042de None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:48.619558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b401ae5-f38b-4ba5-ac39-2a4b4ca042de None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:48.619665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b401ae5-f38b-4ba5-ac39-2a4b4ca042de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:48.684236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-8b401ae5-f38b-4ba5-ac39-2a4b4ca042de None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:37:48.685444 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8b401ae5-f38b-4ba5-ac39-2a4b4ca042de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:48.779747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 661/1331] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:37:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:37:49.066188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af64128b-1ce5-4f43-9f28-c0dd1f21a97d None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:37:49.067766 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-af64128b-1ce5-4f43-9f28-c0dd1f21a97d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:49.168719 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af64128b-1ce5-4f43-9f28-c0dd1f21a97d None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:37:49.241549 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43fbfa01-d40a-42e4-9ba2-5614ae0cfa18 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:37:49.242149 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fbfa01-d40a-42e4-9ba2-5614ae0cfa18 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:49.242428 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fbfa01-d40a-42e4-9ba2-5614ae0cfa18 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:49.242729 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fbfa01-d40a-42e4-9ba2-5614ae0cfa18 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:49.244316 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-43fbfa01-d40a-42e4-9ba2-5614ae0cfa18 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:49.271392 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:37:49.352010 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43fbfa01-d40a-42e4-9ba2-5614ae0cfa18 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:37:49.353384 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43fbfa01-d40a-42e4-9ba2-5614ae0cfa18 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:37:49.371774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 671/1332] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:37:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:37:51.534886 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f222536a-6984-4b57-9308-7b0c67895e87 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:51.535335 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f222536a-6984-4b57-9308-7b0c67895e87 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:51.535426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f222536a-6984-4b57-9308-7b0c67895e87 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:51.537390 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 662/1333] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:37:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:37:51.542668 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79951561-78ba-4fe9-8a6f-ec1177b51344 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:51.543057 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79951561-78ba-4fe9-8a6f-ec1177b51344 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:51.543214 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79951561-78ba-4fe9-8a6f-ec1177b51344 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:51.607411 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-79951561-78ba-4fe9-8a6f-ec1177b51344 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:37:51.608597 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-79951561-78ba-4fe9-8a6f-ec1177b51344 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:51.681515 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:37:51.702975 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 672/1334] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:37:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:37:51.986431 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f222536a-6984-4b57-9308-7b0c67895e87 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:37:51.988214 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f222536a-6984-4b57-9308-7b0c67895e87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:52.030685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:37:52.092365 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f222536a-6984-4b57-9308-7b0c67895e87 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:37:52.163179 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be25096-35fd-4084-80fd-405b180144f1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:37:52.163780 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be25096-35fd-4084-80fd-405b180144f1 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:52.164059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be25096-35fd-4084-80fd-405b180144f1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:52.164329 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be25096-35fd-4084-80fd-405b180144f1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:52.165941 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3be25096-35fd-4084-80fd-405b180144f1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:52.248242 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:37:52.273997 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be25096-35fd-4084-80fd-405b180144f1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:37:52.275472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be25096-35fd-4084-80fd-405b180144f1 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:37:52.293305 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 663/1335] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:37:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:37:55.010987 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff1f417-a91b-4d11-a43b-7f3931fa6c21 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:55.011581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff1f417-a91b-4d11-a43b-7f3931fa6c21 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:55.011581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff1f417-a91b-4d11-a43b-7f3931fa6c21 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:55.012857 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 673/1336] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:37:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:37:55.018151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365d376c-8599-43ce-820e-620116762e9e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:55.018568 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365d376c-8599-43ce-820e-620116762e9e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:55.018733 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365d376c-8599-43ce-820e-620116762e9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:55.083460 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-365d376c-8599-43ce-820e-620116762e9e None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:37:55.084670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-365d376c-8599-43ce-820e-620116762e9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:55.180340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 664/1337] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:37:55 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:37:55.459901 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff1f417-a91b-4d11-a43b-7f3931fa6c21 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:37:55.461442 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-5ff1f417-a91b-4d11-a43b-7f3931fa6c21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:55.561851 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff1f417-a91b-4d11-a43b-7f3931fa6c21 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:37:55.634118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ceee77b-c503-444d-9e69-ff2b028c9aec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:37:55.634743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ceee77b-c503-444d-9e69-ff2b028c9aec service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:37:55.635020 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ceee77b-c503-444d-9e69-ff2b028c9aec service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:37:55.635286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ceee77b-c503-444d-9e69-ff2b028c9aec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:37:55.636847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-6ceee77b-c503-444d-9e69-ff2b028c9aec service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:37:55.671000 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:37:55.741957 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ceee77b-c503-444d-9e69-ff2b028c9aec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:37:55.743433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ceee77b-c503-444d-9e69-ff2b028c9aec service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:37:55.762001 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 674/1338] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:37:55 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:38:00.054119 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c9aea3-b266-4d8c-8475-9dee374b4ed9 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:00.054638 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c9aea3-b266-4d8c-8475-9dee374b4ed9 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:00.055061 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c9aea3-b266-4d8c-8475-9dee374b4ed9 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:00.057073 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 665/1339] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:38:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:00.062862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3b1b03-eb32-4a5e-9044-b5ac25ce9078 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:00.063413 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3b1b03-eb32-4a5e-9044-b5ac25ce9078 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:00.063988 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3b1b03-eb32-4a5e-9044-b5ac25ce9078 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:00.138835 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-ee3b1b03-eb32-4a5e-9044-b5ac25ce9078 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:00.140051 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ee3b1b03-eb32-4a5e-9044-b5ac25ce9078 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:00.240383 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:00.251353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 675/1340] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:38:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:38:00.539554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c9aea3-b266-4d8c-8475-9dee374b4ed9 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:38:00.541220 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-50c9aea3-b266-4d8c-8475-9dee374b4ed9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:00.704169 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:00.758076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c9aea3-b266-4d8c-8475-9dee374b4ed9 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:38:00.832207 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b4bf79-e287-49c4-ad21-d8808c176943 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:38:00.832742 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b4bf79-e287-49c4-ad21-d8808c176943 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:00.833632 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b4bf79-e287-49c4-ad21-d8808c176943 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:00.833997 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b4bf79-e287-49c4-ad21-d8808c176943 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:00.835642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a6b4bf79-e287-49c4-ad21-d8808c176943 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:00.928049 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:00.944930 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b4bf79-e287-49c4-ad21-d8808c176943 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:38:00.946518 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b4bf79-e287-49c4-ad21-d8808c176943 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:38:00.965054 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 666/1341] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:38:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:38:04.108032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9ee711-af4e-4ab2-bcb7-1fcc50247e8e None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:04.108561 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9ee711-af4e-4ab2-bcb7-1fcc50247e8e None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:04.108561 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9ee711-af4e-4ab2-bcb7-1fcc50247e8e None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:04.109843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 676/1342] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:38:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:04.115229 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e2d5fd-2bcb-468f-ab21-d323b11d67e0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:04.115709 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e2d5fd-2bcb-468f-ab21-d323b11d67e0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:04.115803 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e2d5fd-2bcb-468f-ab21-d323b11d67e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:04.183563 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-57e2d5fd-2bcb-468f-ab21-d323b11d67e0 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:04.184780 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-57e2d5fd-2bcb-468f-ab21-d323b11d67e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:04.287052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 667/1343] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:38:04 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:38:04.569936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9ee711-af4e-4ab2-bcb7-1fcc50247e8e None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:38:04.571608 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7a9ee711-af4e-4ab2-bcb7-1fcc50247e8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:04.709050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9ee711-af4e-4ab2-bcb7-1fcc50247e8e None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:38:04.789318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4324cfa-54d9-407e-aa53-df788073e74c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:38:04.789946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4324cfa-54d9-407e-aa53-df788073e74c service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:04.790232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4324cfa-54d9-407e-aa53-df788073e74c service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:04.790505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4324cfa-54d9-407e-aa53-df788073e74c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:04.792166 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-f4324cfa-54d9-407e-aa53-df788073e74c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:04.840234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:04.909259 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4324cfa-54d9-407e-aa53-df788073e74c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:38:04.911492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4324cfa-54d9-407e-aa53-df788073e74c service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:38:04.932341 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 677/1344] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:38:04 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:38:09.210434 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2d461f-6aa4-4cc1-b185-e591884b3cd5 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:09.210932 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2d461f-6aa4-4cc1-b185-e591884b3cd5 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:09.211098 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2d461f-6aa4-4cc1-b185-e591884b3cd5 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:09.212601 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 668/1345] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:38:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:09.218066 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903e0fb4-d16e-4c1f-96f2-d4425e563aa2 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:09.218495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903e0fb4-d16e-4c1f-96f2-d4425e563aa2 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:09.218629 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903e0fb4-d16e-4c1f-96f2-d4425e563aa2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:09.285081 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-903e0fb4-d16e-4c1f-96f2-d4425e563aa2 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:09.286853 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-903e0fb4-d16e-4c1f-96f2-d4425e563aa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:09.385066 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:09.387315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 678/1346] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:38:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:38:09.680103 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2d461f-6aa4-4cc1-b185-e591884b3cd5 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:38:09.681685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9c2d461f-6aa4-4cc1-b185-e591884b3cd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:09.741254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:09.787046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2d461f-6aa4-4cc1-b185-e591884b3cd5 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:38:09.858602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcf53be-5b71-4386-b48d-130304abc89e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:38:09.859212 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcf53be-5b71-4386-b48d-130304abc89e service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:09.859486 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcf53be-5b71-4386-b48d-130304abc89e service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:09.859754 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcf53be-5b71-4386-b48d-130304abc89e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:09.861326 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8fcf53be-5b71-4386-b48d-130304abc89e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:09.963880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:09.967258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fcf53be-5b71-4386-b48d-130304abc89e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:38:09.968675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fcf53be-5b71-4386-b48d-130304abc89e service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:38:09.986317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 669/1347] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:38:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:38:14.929547 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19c83ba-a4f7-4ee3-aa0c-dab4e7fdd5bf None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:14.930086 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19c83ba-a4f7-4ee3-aa0c-dab4e7fdd5bf None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:14.930086 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19c83ba-a4f7-4ee3-aa0c-dab4e7fdd5bf None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:14.931445 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 679/1348] 10.222.0.39 () {54 vars in 921 bytes} [Fri Sep 30 09:38:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:14.937225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1bd474-ea4e-4dfa-ba3b-e141ef0c835a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:14.937660 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1bd474-ea4e-4dfa-ba3b-e141ef0c835a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:14.937759 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1bd474-ea4e-4dfa-ba3b-e141ef0c835a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:15.029887 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9f1bd474-ea4e-4dfa-ba3b-e141ef0c835a None None] MFA Rules not processed for user `49a3011c103b4e48904f876f81d27fba`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:15.031128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9f1bd474-ea4e-4dfa-ba3b-e141ef0c835a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:15.225020 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 670/1349] 10.222.0.39 () {58 vars in 1018 bytes} [Fri Sep 30 09:38:14 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 289 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:38:19.391580 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba6aa85-63a8-4baa-a9ea-ab6858e1405d None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:19.392444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba6aa85-63a8-4baa-a9ea-ab6858e1405d None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:19.392712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba6aa85-63a8-4baa-a9ea-ab6858e1405d None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:19.394044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 680/1350] 10.222.0.39 () {54 vars in 921 bytes} [Fri Sep 30 09:38:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:19.399430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcffdeb-38bc-4e4a-a070-bf76e44a95b4 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:19.400244 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcffdeb-38bc-4e4a-a070-bf76e44a95b4 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:19.400717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcffdeb-38bc-4e4a-a070-bf76e44a95b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:19.461162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:19.490631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-dbcffdeb-38bc-4e4a-a070-bf76e44a95b4 None None] MFA Rules not processed for user `49a3011c103b4e48904f876f81d27fba`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:19.492544 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-dbcffdeb-38bc-4e4a-a070-bf76e44a95b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:19.609042 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 671/1351] 10.222.0.39 () {58 vars in 1018 bytes} [Fri Sep 30 09:38:19 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:38:20.854107 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8503fa-2332-4d30-975b-90c9573a60a5 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:20.854688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8503fa-2332-4d30-975b-90c9573a60a5 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:20.854688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8503fa-2332-4d30-975b-90c9573a60a5 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:20.856275 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 681/1352] 10.222.0.39 () {54 vars in 921 bytes} [Fri Sep 30 09:38:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:20.861478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28758c24-c832-49a8-8f8f-b8b2091c93fa None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:20.861934 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28758c24-c832-49a8-8f8f-b8b2091c93fa None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:20.862040 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28758c24-c832-49a8-8f8f-b8b2091c93fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:20.944395 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-28758c24-c832-49a8-8f8f-b8b2091c93fa None None] MFA Rules not processed for user `49a3011c103b4e48904f876f81d27fba`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:20.945609 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-28758c24-c832-49a8-8f8f-b8b2091c93fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:20.988668 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:21.055644 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 672/1353] 10.222.0.39 () {58 vars in 1018 bytes} [Fri Sep 30 09:38:20 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:38:24.640022 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab0df91-b5c9-4483-86bc-f15b8752eb62 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:24.640666 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab0df91-b5c9-4483-86bc-f15b8752eb62 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:24.640666 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab0df91-b5c9-4483-86bc-f15b8752eb62 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:24.641796 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 682/1354] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:38:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:24.647777 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce135604-c9c2-4bc5-81e6-ca1307be50db None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:24.654998 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce135604-c9c2-4bc5-81e6-ca1307be50db None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:24.654998 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce135604-c9c2-4bc5-81e6-ca1307be50db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:24.718601 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ce135604-c9c2-4bc5-81e6-ca1307be50db None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:24.719674 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ce135604-c9c2-4bc5-81e6-ca1307be50db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:24.818044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 673/1355] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:38:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:38:25.119188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928aef17-9a4e-480a-b7e3-9b2beca649a6 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:25.119903 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928aef17-9a4e-480a-b7e3-9b2beca649a6 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:25.120392 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928aef17-9a4e-480a-b7e3-9b2beca649a6 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:25.122014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 683/1356] 10.222.0.39 () {54 vars in 961 bytes} [Fri Sep 30 09:38:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:25.129905 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd59ec8f-2166-4d62-9039-a0be30af5096 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:25.134747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd59ec8f-2166-4d62-9039-a0be30af5096 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:25.134747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd59ec8f-2166-4d62-9039-a0be30af5096 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:25.147746 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:25.210120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-cd59ec8f-2166-4d62-9039-a0be30af5096 None None] MFA Rules not processed for user `7ff68fb87700423ab53c90520a6bd3b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:25.211398 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-cd59ec8f-2166-4d62-9039-a0be30af5096 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:25.344757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 674/1357] 10.222.0.39 () {58 vars in 1058 bytes} [Fri Sep 30 09:38:25 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:38:25.353826 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928aef17-9a4e-480a-b7e3-9b2beca649a6 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:38:25.355426 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-928aef17-9a4e-480a-b7e3-9b2beca649a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:25.523882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928aef17-9a4e-480a-b7e3-9b2beca649a6 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:38:25.603116 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:25.645146 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de716200-f816-4097-acfe-d74e46fde578 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:38:25.645822 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de716200-f816-4097-acfe-d74e46fde578 service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:25.646033 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de716200-f816-4097-acfe-d74e46fde578 service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:25.646277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de716200-f816-4097-acfe-d74e46fde578 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:25.647903 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-de716200-f816-4097-acfe-d74e46fde578 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:25.745981 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de716200-f816-4097-acfe-d74e46fde578 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:38:25.747636 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de716200-f816-4097-acfe-d74e46fde578 service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:38:25.767839 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 684/1358] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:38:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:38:28.168665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f0b007-0b96-4648-ba16-718146ae7d99 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:28.169188 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f0b007-0b96-4648-ba16-718146ae7d99 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:28.169188 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f0b007-0b96-4648-ba16-718146ae7d99 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:28.170837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 675/1359] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:38:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:28.176748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3c0a5d-7291-4e34-91d5-61cc94dc2f6a None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:28.177568 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3c0a5d-7291-4e34-91d5-61cc94dc2f6a None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:28.178022 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3c0a5d-7291-4e34-91d5-61cc94dc2f6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:28.283633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-9d3c0a5d-7291-4e34-91d5-61cc94dc2f6a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:28.285674 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-9d3c0a5d-7291-4e34-91d5-61cc94dc2f6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:28.326102 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:28.391430 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 685/1360] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:38:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:38:28.683014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f0b007-0b96-4648-ba16-718146ae7d99 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:38:28.684714 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-52f0b007-0b96-4648-ba16-718146ae7d99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:28.758542 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:28.856351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f0b007-0b96-4648-ba16-718146ae7d99 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:38:28.986204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36815de-a269-42c4-abe7-7c765e881c63 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:38:28.986850 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36815de-a269-42c4-abe7-7c765e881c63 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:28.987353 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36815de-a269-42c4-abe7-7c765e881c63 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:28.987466 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36815de-a269-42c4-abe7-7c765e881c63 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:28.989506 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e36815de-a269-42c4-abe7-7c765e881c63 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:29.015251 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:29.084372 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36815de-a269-42c4-abe7-7c765e881c63 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:38:29.085985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36815de-a269-42c4-abe7-7c765e881c63 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:38:29.104817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 676/1361] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:38:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:38:31.703329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f22be4-a6b7-44c4-951c-3764ff262fa4 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:31.703862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f22be4-a6b7-44c4-951c-3764ff262fa4 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:31.703862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f22be4-a6b7-44c4-951c-3764ff262fa4 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:31.705298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 686/1362] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:38:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:31.711076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f65db3-737f-4e42-836c-cffaf5c9555f None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:31.711506 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f65db3-737f-4e42-836c-cffaf5c9555f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:31.711725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f65db3-737f-4e42-836c-cffaf5c9555f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:31.784691 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-13f65db3-737f-4e42-836c-cffaf5c9555f None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:31.785926 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-13f65db3-737f-4e42-836c-cffaf5c9555f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:31.894307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:31.906996 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 677/1363] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:38:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:38:32.189237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cf13db-3e5e-48c4-a91d-7384786e1cd1 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:32.189599 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cf13db-3e5e-48c4-a91d-7384786e1cd1 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:32.189961 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cf13db-3e5e-48c4-a91d-7384786e1cd1 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:32.191931 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 687/1364] 10.222.0.39 () {54 vars in 961 bytes} [Fri Sep 30 09:38:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:32.197592 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce228c1-0e09-476c-9c33-0e5effe2bd7e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:32.197955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce228c1-0e09-476c-9c33-0e5effe2bd7e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:32.198312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce228c1-0e09-476c-9c33-0e5effe2bd7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:32.283714 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-dce228c1-0e09-476c-9c33-0e5effe2bd7e None None] MFA Rules not processed for user `7ff68fb87700423ab53c90520a6bd3b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:32.284967 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-dce228c1-0e09-476c-9c33-0e5effe2bd7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:32.410626 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 678/1365] 10.222.0.39 () {58 vars in 1058 bytes} [Fri Sep 30 09:38:32 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:38:32.417371 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cf13db-3e5e-48c4-a91d-7384786e1cd1 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:38:32.418946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b4cf13db-3e5e-48c4-a91d-7384786e1cd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:32.571650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cf13db-3e5e-48c4-a91d-7384786e1cd1 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:38:32.592991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:32.690121 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafb6e65-589f-4f24-834b-94ae01679b67 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:38:32.691352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafb6e65-589f-4f24-834b-94ae01679b67 service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:32.691634 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafb6e65-589f-4f24-834b-94ae01679b67 service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:32.691907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafb6e65-589f-4f24-834b-94ae01679b67 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:32.693684 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-eafb6e65-589f-4f24-834b-94ae01679b67 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:32.792513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafb6e65-589f-4f24-834b-94ae01679b67 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:38:32.794968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafb6e65-589f-4f24-834b-94ae01679b67 service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:38:32.813333 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff632348-51a7-4594-88d7-daca92e22353 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:32.813837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff632348-51a7-4594-88d7-daca92e22353 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:32.813957 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff632348-51a7-4594-88d7-daca92e22353 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:32.815271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 679/1366] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:38:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:32.818328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 688/1367] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:38:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:38:32.820763 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f8be38-e4e5-4338-9ce6-459e7ffa8e93 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:32.821052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f8be38-e4e5-4338-9ce6-459e7ffa8e93 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:32.821331 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f8be38-e4e5-4338-9ce6-459e7ffa8e93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:32.883146 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:32.890861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b0f8be38-e4e5-4338-9ce6-459e7ffa8e93 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:32.892085 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b0f8be38-e4e5-4338-9ce6-459e7ffa8e93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:33.004140 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 680/1368] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:38:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:38:33.340141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb38dc46-13d2-4eb6-bac0-6e5cf6630920 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:33.340141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb38dc46-13d2-4eb6-bac0-6e5cf6630920 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:33.340141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb38dc46-13d2-4eb6-bac0-6e5cf6630920 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:33.341484 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 689/1369] 10.222.0.39 () {54 vars in 963 bytes} [Fri Sep 30 09:38:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:33.346606 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf57a02-520c-41c3-9f5a-d4233e1bcd47 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:33.347439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf57a02-520c-41c3-9f5a-d4233e1bcd47 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:33.347714 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf57a02-520c-41c3-9f5a-d4233e1bcd47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:33.429777 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-6cf57a02-520c-41c3-9f5a-d4233e1bcd47 None None] MFA Rules not processed for user `4e2742f567444dff88ed919be52217de`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:33.431032 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6cf57a02-520c-41c3-9f5a-d4233e1bcd47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:33.511919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:33.520410 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 681/1370] 10.222.0.39 () {58 vars in 1060 bytes} [Fri Sep 30 09:38:33 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:38:33.528393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb38dc46-13d2-4eb6-bac0-6e5cf6630920 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:38:33.529910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-cb38dc46-13d2-4eb6-bac0-6e5cf6630920 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:33.563116 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:33.645853 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb38dc46-13d2-4eb6-bac0-6e5cf6630920 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:38:33.733699 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411d0609-6861-4fe0-952d-e9b981d08d92 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:38:33.734380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411d0609-6861-4fe0-952d-e9b981d08d92 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:33.734691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411d0609-6861-4fe0-952d-e9b981d08d92 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:33.734961 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411d0609-6861-4fe0-952d-e9b981d08d92 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:33.736550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-411d0609-6861-4fe0-952d-e9b981d08d92 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:33.822422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:33.861200 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411d0609-6861-4fe0-952d-e9b981d08d92 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:38:33.870004 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411d0609-6861-4fe0-952d-e9b981d08d92 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:38:33.882688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 690/1371] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:38:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:38:35.202092 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacff0e8-1a97-4ba5-96c6-80fce2f4029f None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:35.202574 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacff0e8-1a97-4ba5-96c6-80fce2f4029f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:35.202702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacff0e8-1a97-4ba5-96c6-80fce2f4029f None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:35.204037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 682/1372] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:38:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:35.209817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce397e40-b009-4301-ac13-30fd45f7cbb0 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:35.210225 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce397e40-b009-4301-ac13-30fd45f7cbb0 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:35.210487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce397e40-b009-4301-ac13-30fd45f7cbb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:35.281613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-ce397e40-b009-4301-ac13-30fd45f7cbb0 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:35.282874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ce397e40-b009-4301-ac13-30fd45f7cbb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:35.380899 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 691/1373] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:38:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:38:35.672571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacff0e8-1a97-4ba5-96c6-80fce2f4029f None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:38:35.674155 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-cacff0e8-1a97-4ba5-96c6-80fce2f4029f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:35.828427 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacff0e8-1a97-4ba5-96c6-80fce2f4029f None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:38:35.900536 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:35.949880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5e45ed-5e0f-4cd4-8826-01389ee470f7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:38:35.950497 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5e45ed-5e0f-4cd4-8826-01389ee470f7 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:35.950805 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5e45ed-5e0f-4cd4-8826-01389ee470f7 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:35.951072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5e45ed-5e0f-4cd4-8826-01389ee470f7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:35.952694 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2c5e45ed-5e0f-4cd4-8826-01389ee470f7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:36.050278 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5e45ed-5e0f-4cd4-8826-01389ee470f7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:38:36.051918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5e45ed-5e0f-4cd4-8826-01389ee470f7 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:38:36.071147 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 683/1374] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:38:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:38:38.418925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4b6533-bb3e-462b-98d5-90f8a6decd79 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:38.419445 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4b6533-bb3e-462b-98d5-90f8a6decd79 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:38.419445 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4b6533-bb3e-462b-98d5-90f8a6decd79 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:38.420884 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 692/1375] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:38:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:38.426241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865594f6-7415-4944-a6e3-3418f8a62b1d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:38.426683 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865594f6-7415-4944-a6e3-3418f8a62b1d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:38.426999 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865594f6-7415-4944-a6e3-3418f8a62b1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:38.494731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-865594f6-7415-4944-a6e3-3418f8a62b1d None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:38.496088 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-865594f6-7415-4944-a6e3-3418f8a62b1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:38.527732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:38.597768 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 684/1376] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:38:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:38:38.889921 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4b6533-bb3e-462b-98d5-90f8a6decd79 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:38:38.891480 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-5f4b6533-bb3e-462b-98d5-90f8a6decd79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:38.930766 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:39.042965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4b6533-bb3e-462b-98d5-90f8a6decd79 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:38:39.160761 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:39.165993 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0281fb2-c709-45b2-900a-2f4ed0dd42c1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:38:39.166616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0281fb2-c709-45b2-900a-2f4ed0dd42c1 service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:39.166904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0281fb2-c709-45b2-900a-2f4ed0dd42c1 service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:39.167176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0281fb2-c709-45b2-900a-2f4ed0dd42c1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:39.168748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c0281fb2-c709-45b2-900a-2f4ed0dd42c1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:39.269329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0281fb2-c709-45b2-900a-2f4ed0dd42c1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:38:39.271149 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0281fb2-c709-45b2-900a-2f4ed0dd42c1 service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:38:39.290219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 693/1377] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:38:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:38:41.656513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f6e79f-c102-42f1-b0f2-5ab9069de388 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:41.657057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f6e79f-c102-42f1-b0f2-5ab9069de388 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:41.657057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f6e79f-c102-42f1-b0f2-5ab9069de388 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:41.658414 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 685/1378] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:38:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:41.664319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18ad570-de56-4c5b-897f-bec590f9ca1a None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:41.664763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18ad570-de56-4c5b-897f-bec590f9ca1a None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:41.664877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18ad570-de56-4c5b-897f-bec590f9ca1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:41.739521 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-d18ad570-de56-4c5b-897f-bec590f9ca1a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:41.740774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d18ad570-de56-4c5b-897f-bec590f9ca1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:41.805965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:41.840847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 694/1379] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:38:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:38:42.139259 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f6e79f-c102-42f1-b0f2-5ab9069de388 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:38:42.140795 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c1f6e79f-c102-42f1-b0f2-5ab9069de388 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:42.292890 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f6e79f-c102-42f1-b0f2-5ab9069de388 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:38:42.306757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:42.425609 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084f6e5d-6566-460d-b6d5-5e5db3cb2c09 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:38:42.427616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084f6e5d-6566-460d-b6d5-5e5db3cb2c09 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:42.427963 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084f6e5d-6566-460d-b6d5-5e5db3cb2c09 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:42.428382 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084f6e5d-6566-460d-b6d5-5e5db3cb2c09 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:42.430463 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-084f6e5d-6566-460d-b6d5-5e5db3cb2c09 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:42.521694 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084f6e5d-6566-460d-b6d5-5e5db3cb2c09 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:38:42.523330 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084f6e5d-6566-460d-b6d5-5e5db3cb2c09 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:38:42.541649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 686/1380] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:38:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:38:44.860166 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b853cc0a-da30-414b-b134-48fd6abf3219 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:44.860706 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b853cc0a-da30-414b-b134-48fd6abf3219 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:44.860706 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b853cc0a-da30-414b-b134-48fd6abf3219 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:44.862059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 695/1381] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:38:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:44.869162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89e762d-b945-4b34-94a6-88490fe44c81 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:44.869596 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89e762d-b945-4b34-94a6-88490fe44c81 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:44.869905 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89e762d-b945-4b34-94a6-88490fe44c81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:44.898331 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:44.955787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f89e762d-b945-4b34-94a6-88490fe44c81 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:44.957952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f89e762d-b945-4b34-94a6-88490fe44c81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:45.086820 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 687/1382] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:38:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:38:45.447179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b853cc0a-da30-414b-b134-48fd6abf3219 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:38:45.448973 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b853cc0a-da30-414b-b134-48fd6abf3219 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:45.616188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b853cc0a-da30-414b-b134-48fd6abf3219 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:38:45.683333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:45.748123 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb5f2ab-0593-408c-bdad-5580b21922ac service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:38:45.748761 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb5f2ab-0593-408c-bdad-5580b21922ac service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:45.751941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb5f2ab-0593-408c-bdad-5580b21922ac service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:45.751941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb5f2ab-0593-408c-bdad-5580b21922ac service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:45.751941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-6fb5f2ab-0593-408c-bdad-5580b21922ac service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:45.844721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb5f2ab-0593-408c-bdad-5580b21922ac service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:38:45.846333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb5f2ab-0593-408c-bdad-5580b21922ac service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:38:45.864356 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 696/1383] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:38:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:38:48.020750 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0569b9-d9c8-49cc-9935-a22cebf7e3c7 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:48.021317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0569b9-d9c8-49cc-9935-a22cebf7e3c7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:48.021317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0569b9-d9c8-49cc-9935-a22cebf7e3c7 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:48.022939 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 688/1384] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:38:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:48.028599 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc5037c-63de-44dd-a106-04f3977780b5 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:48.029142 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc5037c-63de-44dd-a106-04f3977780b5 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:48.029237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc5037c-63de-44dd-a106-04f3977780b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:48.094354 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-efc5037c-63de-44dd-a106-04f3977780b5 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:48.095583 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-efc5037c-63de-44dd-a106-04f3977780b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:48.104471 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:48.198935 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 697/1385] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:38:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:38:48.490078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0569b9-d9c8-49cc-9935-a22cebf7e3c7 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:38:48.491681 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ec0569b9-d9c8-49cc-9935-a22cebf7e3c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:48.596796 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:48.656421 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0569b9-d9c8-49cc-9935-a22cebf7e3c7 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:38:48.768752 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d32c0e-74e7-47ed-87e0-4765ac473763 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:38:48.769819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d32c0e-74e7-47ed-87e0-4765ac473763 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:48.770100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d32c0e-74e7-47ed-87e0-4765ac473763 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:48.770370 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d32c0e-74e7-47ed-87e0-4765ac473763 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:48.772078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-61d32c0e-74e7-47ed-87e0-4765ac473763 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:48.832491 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:48.862159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d32c0e-74e7-47ed-87e0-4765ac473763 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:38:48.864595 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d32c0e-74e7-47ed-87e0-4765ac473763 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:38:48.883087 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 689/1386] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:38:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:38:49.290481 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da47aae-94c5-45ac-a5e7-ab45bcaf1ec9 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:49.291065 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da47aae-94c5-45ac-a5e7-ab45bcaf1ec9 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:49.291065 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da47aae-94c5-45ac-a5e7-ab45bcaf1ec9 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:49.292382 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 698/1387] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:38:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:49.297540 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97592a70-3b9f-47f4-afdd-a557e31d2f78 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:49.297973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97592a70-3b9f-47f4-afdd-a557e31d2f78 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:49.298076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97592a70-3b9f-47f4-afdd-a557e31d2f78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:49.373050 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-97592a70-3b9f-47f4-afdd-a557e31d2f78 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:49.374166 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-97592a70-3b9f-47f4-afdd-a557e31d2f78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:49.470701 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 690/1388] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:38:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:38:49.770794 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5387a693-47ed-4439-8c24-7d8f859d0a7b None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:49.771147 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5387a693-47ed-4439-8c24-7d8f859d0a7b None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:49.771501 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5387a693-47ed-4439-8c24-7d8f859d0a7b None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:49.772936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 699/1389] 10.222.0.39 () {54 vars in 962 bytes} [Fri Sep 30 09:38:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:49.778171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5341e32-df44-4880-83ff-16c038c40264 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:49.778459 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5341e32-df44-4880-83ff-16c038c40264 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:49.778777 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5341e32-df44-4880-83ff-16c038c40264 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:49.825012 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:49.856991 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c5341e32-df44-4880-83ff-16c038c40264 None None] MFA Rules not processed for user `bf907d4033064fb8a2707b1bbb0a180b`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:49.858199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c5341e32-df44-4880-83ff-16c038c40264 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:49.936126 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 691/1390] 10.222.0.39 () {58 vars in 1059 bytes} [Fri Sep 30 09:38:49 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:38:49.945247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5387a693-47ed-4439-8c24-7d8f859d0a7b None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:38:49.946836 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-5387a693-47ed-4439-8c24-7d8f859d0a7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:50.050188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5387a693-47ed-4439-8c24-7d8f859d0a7b None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:38:50.088312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:50.126570 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d714a39-1e50-4cd3-8b0d-23dc11329e8d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf907d4033064fb8a2707b1bbb0a180b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:38:50.127191 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d714a39-1e50-4cd3-8b0d-23dc11329e8d service glance] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:50.127465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d714a39-1e50-4cd3-8b0d-23dc11329e8d service glance] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:50.127733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d714a39-1e50-4cd3-8b0d-23dc11329e8d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:50.129333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-3d714a39-1e50-4cd3-8b0d-23dc11329e8d service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:50.239872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d714a39-1e50-4cd3-8b0d-23dc11329e8d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:38:50.241465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d714a39-1e50-4cd3-8b0d-23dc11329e8d service glance] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:38:50.260396 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 700/1391] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:38:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:38:51.164541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c4300f-6725-4af8-b019-fe4fdddd8712 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:51.164995 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c4300f-6725-4af8-b019-fe4fdddd8712 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:51.165172 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c4300f-6725-4af8-b019-fe4fdddd8712 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:51.177294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb31741-7a86-4f86-95df-4dc7513e2bab None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:51.177294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb31741-7a86-4f86-95df-4dc7513e2bab None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:51.177294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb31741-7a86-4f86-95df-4dc7513e2bab None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:51.177294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 701/1392] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:38:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:38:51.181921 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5caecb6-152a-4f82-a145-0f4f0219d793 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:51.182203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5caecb6-152a-4f82-a145-0f4f0219d793 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:51.182474 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5caecb6-152a-4f82-a145-0f4f0219d793 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:51.236676 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-54c4300f-6725-4af8-b019-fe4fdddd8712 None None] MFA Rules not processed for user `7d6e0545a40a436d8c63820b795551a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:51.237894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-54c4300f-6725-4af8-b019-fe4fdddd8712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:51.251164 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-d5caecb6-152a-4f82-a145-0f4f0219d793 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:51.252394 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d5caecb6-152a-4f82-a145-0f4f0219d793 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:51.272504 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:51.318432 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:51.325692 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 692/1393] 10.222.0.39 () {58 vars in 1009 bytes} [Fri Sep 30 09:38:51 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:38:51.353656 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 702/1394] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:38:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:38:51.370943 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4289f72a-e1f6-4121-b782-c5c0191dc961 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:51.371257 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4289f72a-e1f6-4121-b782-c5c0191dc961 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:51.371565 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4289f72a-e1f6-4121-b782-c5c0191dc961 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:51.372920 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 693/1395] 10.222.0.39 () {54 vars in 943 bytes} [Fri Sep 30 09:38:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:51.379606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8246cc0-cfba-4abf-a345-7abd483094af None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:51.379899 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8246cc0-cfba-4abf-a345-7abd483094af None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:51.380177 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8246cc0-cfba-4abf-a345-7abd483094af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:51.460983 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-d8246cc0-cfba-4abf-a345-7abd483094af None None] MFA Rules not processed for user `0e5ad921e6fa41c297696ec7279f0fc4`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:51.462187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d8246cc0-cfba-4abf-a345-7abd483094af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:51.544316 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:51.571041 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 703/1396] 10.222.0.39 () {58 vars in 1040 bytes} [Fri Sep 30 09:38:51 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:38:51.581130 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4289f72a-e1f6-4121-b782-c5c0191dc961 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:38:51.582823 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4289f72a-e1f6-4121-b782-c5c0191dc961 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:51.643658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8246cc0-cfba-4abf-a345-7abd483094af None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:38:51.644940 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d8246cc0-cfba-4abf-a345-7abd483094af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:51.729052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4289f72a-e1f6-4121-b782-c5c0191dc961 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:38:51.815303 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:51.841076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37de8ddb-e3f5-4955-af7a-a01951315a59 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e5ad921e6fa41c297696ec7279f0fc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:38:51.841773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37de8ddb-e3f5-4955-af7a-a01951315a59 service swift] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:51.842058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37de8ddb-e3f5-4955-af7a-a01951315a59 service swift] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:51.842330 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37de8ddb-e3f5-4955-af7a-a01951315a59 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:51.844113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-37de8ddb-e3f5-4955-af7a-a01951315a59 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:51.861641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8246cc0-cfba-4abf-a345-7abd483094af None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:38:51.928742 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37de8ddb-e3f5-4955-af7a-a01951315a59 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:38:51.930360 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37de8ddb-e3f5-4955-af7a-a01951315a59 service swift] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:38:51.938569 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 694/1397] 10.222.0.39 () {60 vars in 1384 bytes} [Fri Sep 30 09:38:51 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 30 09:38:51.967837 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:51.978164 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3b9f4b-73d9-4df6-b2b8-c47f5b270d93 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:38:51.978803 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3b9f4b-73d9-4df6-b2b8-c47f5b270d93 service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:51.982302 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3b9f4b-73d9-4df6-b2b8-c47f5b270d93 service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:51.982302 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3b9f4b-73d9-4df6-b2b8-c47f5b270d93 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:51.982302 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ca3b9f4b-73d9-4df6-b2b8-c47f5b270d93 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:52.069832 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3b9f4b-73d9-4df6-b2b8-c47f5b270d93 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:38:52.071409 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3b9f4b-73d9-4df6-b2b8-c47f5b270d93 service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:38:52.089028 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 704/1398] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:38:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 446 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:38:54.499600 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eecd3c9-a514-4068-9fc6-e3c6117822d1 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:54.500752 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eecd3c9-a514-4068-9fc6-e3c6117822d1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:54.501290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eecd3c9-a514-4068-9fc6-e3c6117822d1 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:54.503363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 695/1399] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:38:54 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:54.510783 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d0b5c0-90da-4e4f-9c60-ec7c3b5d2f3a None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:54.511336 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d0b5c0-90da-4e4f-9c60-ec7c3b5d2f3a None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:54.511336 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d0b5c0-90da-4e4f-9c60-ec7c3b5d2f3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:54.593617 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-35d0b5c0-90da-4e4f-9c60-ec7c3b5d2f3a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:54.595235 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-35d0b5c0-90da-4e4f-9c60-ec7c3b5d2f3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:54.677020 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:54.716347 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 705/1400] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:38:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:38:55.017903 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eecd3c9-a514-4068-9fc6-e3c6117822d1 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:38:55.020486 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0eecd3c9-a514-4068-9fc6-e3c6117822d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:55.150158 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:55.197057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eecd3c9-a514-4068-9fc6-e3c6117822d1 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:38:55.332110 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27904fe7-b94e-4fc7-93bc-1a6c5344c1ff service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:38:55.332747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27904fe7-b94e-4fc7-93bc-1a6c5344c1ff service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:55.333090 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27904fe7-b94e-4fc7-93bc-1a6c5344c1ff service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:55.333434 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27904fe7-b94e-4fc7-93bc-1a6c5344c1ff service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:55.335052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-27904fe7-b94e-4fc7-93bc-1a6c5344c1ff service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:55.427610 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:55.442896 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27904fe7-b94e-4fc7-93bc-1a6c5344c1ff service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:38:55.444542 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27904fe7-b94e-4fc7-93bc-1a6c5344c1ff service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:38:55.464488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 696/1401] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:38:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 447 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:38:57.787174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9435e21e-9e7b-4154-a87a-2810da86d210 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:57.788182 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9435e21e-9e7b-4154-a87a-2810da86d210 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:57.788550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9435e21e-9e7b-4154-a87a-2810da86d210 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:57.789977 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 706/1402] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:38:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:38:57.795849 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972be994-7193-4b4d-970f-56ed6dc4cc85 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:57.796277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972be994-7193-4b4d-970f-56ed6dc4cc85 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:57.796379 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972be994-7193-4b4d-970f-56ed6dc4cc85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:57.869796 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-972be994-7193-4b4d-970f-56ed6dc4cc85 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:38:57.871068 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-972be994-7193-4b4d-970f-56ed6dc4cc85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:57.976455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 697/1403] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:38:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:38:58.270152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9435e21e-9e7b-4154-a87a-2810da86d210 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:38:58.271748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-9435e21e-9e7b-4154-a87a-2810da86d210 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:58.433547 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9435e21e-9e7b-4154-a87a-2810da86d210 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:38:58.489894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:38:58.559180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248b871b-66e6-4946-9362-84dd0951d78a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:38:58.559904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248b871b-66e6-4946-9362-84dd0951d78a service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:38:58.560183 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248b871b-66e6-4946-9362-84dd0951d78a service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:38:58.560452 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248b871b-66e6-4946-9362-84dd0951d78a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:38:58.562068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-248b871b-66e6-4946-9362-84dd0951d78a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:38:58.668104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248b871b-66e6-4946-9362-84dd0951d78a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:38:58.669858 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248b871b-66e6-4946-9362-84dd0951d78a service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:38:58.694923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 707/1404] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:38:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:39:00.977531 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1134d7-ede8-4507-800a-d60b84eba2c6 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:00.978079 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1134d7-ede8-4507-800a-d60b84eba2c6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:00.978079 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1134d7-ede8-4507-800a-d60b84eba2c6 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:00.983265 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 698/1405] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:39:00 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:00.984689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b165f6b-644d-4960-bbdc-afe3e52f4897 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:00.985101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b165f6b-644d-4960-bbdc-afe3e52f4897 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:00.985244 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b165f6b-644d-4960-bbdc-afe3e52f4897 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:01.058421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-2b165f6b-644d-4960-bbdc-afe3e52f4897 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:01.059809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-2b165f6b-644d-4960-bbdc-afe3e52f4897 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:01.063122 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:01.180924 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 708/1406] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:39:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:39:01.469953 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1134d7-ede8-4507-800a-d60b84eba2c6 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:39:01.472170 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fc1134d7-ede8-4507-800a-d60b84eba2c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:01.540710 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:01.631880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1134d7-ede8-4507-800a-d60b84eba2c6 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:39:01.760375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a89c5b1-8aa8-4191-b34a-01e8f4f2accf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:39:01.760980 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a89c5b1-8aa8-4191-b34a-01e8f4f2accf service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:01.761253 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a89c5b1-8aa8-4191-b34a-01e8f4f2accf service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:01.761516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a89c5b1-8aa8-4191-b34a-01e8f4f2accf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:01.763130 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0a89c5b1-8aa8-4191-b34a-01e8f4f2accf service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:01.788276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:01.875014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a89c5b1-8aa8-4191-b34a-01e8f4f2accf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:39:01.876609 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a89c5b1-8aa8-4191-b34a-01e8f4f2accf service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:39:01.897843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 699/1407] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:39:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:39:11.179502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=41 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:11.900961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=37 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:15.478979 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea649fe4-ecb0-4392-982c-55654209c0f5 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:15.479524 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea649fe4-ecb0-4392-982c-55654209c0f5 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:15.479524 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea649fe4-ecb0-4392-982c-55654209c0f5 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:15.480874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 709/1408] 10.222.0.39 () {54 vars in 918 bytes} [Fri Sep 30 09:39:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:15.486026 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56635812-e03e-450a-aa51-673302432424 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:15.486440 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56635812-e03e-450a-aa51-673302432424 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:15.486813 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56635812-e03e-450a-aa51-673302432424 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:15.561379 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-56635812-e03e-450a-aa51-673302432424 None None] MFA Rules not processed for user `49a3011c103b4e48904f876f81d27fba`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:15.562636 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-56635812-e03e-450a-aa51-673302432424 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:15.669220 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 700/1409] 10.222.0.39 () {58 vars in 1015 bytes} [Fri Sep 30 09:39:15 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:39:15.683469 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea90f0c0-15c2-4ff6-b94f-95b8708cdab1 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:15.683746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea90f0c0-15c2-4ff6-b94f-95b8708cdab1 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:15.684025 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea90f0c0-15c2-4ff6-b94f-95b8708cdab1 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:15.685350 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 710/1410] 10.222.0.39 () {54 vars in 918 bytes} [Fri Sep 30 09:39:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:15.689920 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c4304e-d9e8-4f3b-9c1e-325247e03d8f None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:15.690202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c4304e-d9e8-4f3b-9c1e-325247e03d8f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:15.690475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c4304e-d9e8-4f3b-9c1e-325247e03d8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:15.762384 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:15.764794 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e4c4304e-d9e8-4f3b-9c1e-325247e03d8f None None] MFA Rules not processed for user `7ff68fb87700423ab53c90520a6bd3b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:15.766017 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e4c4304e-d9e8-4f3b-9c1e-325247e03d8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:15.879989 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 701/1411] 10.222.0.39 () {58 vars in 1015 bytes} [Fri Sep 30 09:39:15 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:39:18.432198 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de095e3-885c-4b53-b790-ad3882f98e8a None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:18.432756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de095e3-885c-4b53-b790-ad3882f98e8a None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:18.432756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de095e3-885c-4b53-b790-ad3882f98e8a None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:18.434071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 711/1412] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 30 09:39:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:18.440010 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6880675f-339a-41d7-b02f-e70dca9971d7 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:18.440429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6880675f-339a-41d7-b02f-e70dca9971d7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:18.440555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6880675f-339a-41d7-b02f-e70dca9971d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:18.506804 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-6880675f-339a-41d7-b02f-e70dca9971d7 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:18.508004 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6880675f-339a-41d7-b02f-e70dca9971d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:18.561652 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:18.603149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 702/1413] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 30 09:39:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:39:18.895543 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de095e3-885c-4b53-b790-ad3882f98e8a None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:39:18.897167 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0de095e3-885c-4b53-b790-ad3882f98e8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:19.040422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de095e3-885c-4b53-b790-ad3882f98e8a None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:39:19.111437 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:19.151591 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae23da6-bb1c-48ce-a678-b43a923aaa30 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:39:19.152170 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae23da6-bb1c-48ce-a678-b43a923aaa30 service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:19.152453 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae23da6-bb1c-48ce-a678-b43a923aaa30 service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:19.152933 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae23da6-bb1c-48ce-a678-b43a923aaa30 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:19.155123 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-9ae23da6-bb1c-48ce-a678-b43a923aaa30 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:19.243951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae23da6-bb1c-48ce-a678-b43a923aaa30 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:39:19.245553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae23da6-bb1c-48ce-a678-b43a923aaa30 service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:39:19.263887 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 712/1414] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:39:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:39:20.397185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bcc17a-56e9-4f66-a186-f4c5116fcc2c None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:20.397798 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bcc17a-56e9-4f66-a186-f4c5116fcc2c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:20.397798 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bcc17a-56e9-4f66-a186-f4c5116fcc2c None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:20.399123 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 703/1415] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 30 09:39:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:20.405379 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b3df61-cdf8-489c-b99c-0a0d82e00a7c None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:20.405811 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b3df61-cdf8-489c-b99c-0a0d82e00a7c None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:20.405918 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b3df61-cdf8-489c-b99c-0a0d82e00a7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:20.479644 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-95b3df61-cdf8-489c-b99c-0a0d82e00a7c None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:20.480852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-95b3df61-cdf8-489c-b99c-0a0d82e00a7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:20.507344 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:20.576141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 713/1416] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 30 09:39:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:39:20.725844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bcc17a-56e9-4f66-a186-f4c5116fcc2c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:39:20.727451 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-21bcc17a-56e9-4f66-a186-f4c5116fcc2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:20.872105 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bcc17a-56e9-4f66-a186-f4c5116fcc2c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:39:20.913111 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:20.985213 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f90f298-8e16-4683-be3e-8e6608c6a6ca service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:39:20.985816 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f90f298-8e16-4683-be3e-8e6608c6a6ca service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:20.986613 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f90f298-8e16-4683-be3e-8e6608c6a6ca service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:20.986897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f90f298-8e16-4683-be3e-8e6608c6a6ca service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:20.988650 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5f90f298-8e16-4683-be3e-8e6608c6a6ca service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:21.079247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f90f298-8e16-4683-be3e-8e6608c6a6ca service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:39:21.080843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f90f298-8e16-4683-be3e-8e6608c6a6ca service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:39:21.099130 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 704/1417] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:39:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:39:22.228534 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80e2db3-1234-4983-b356-42c5337de103 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:22.229018 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80e2db3-1234-4983-b356-42c5337de103 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:22.229125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80e2db3-1234-4983-b356-42c5337de103 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:22.230408 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 714/1418] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 30 09:39:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:22.237327 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b27636-e049-4511-9b85-6ef6f519b960 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:22.237775 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b27636-e049-4511-9b85-6ef6f519b960 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:22.237872 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b27636-e049-4511-9b85-6ef6f519b960 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:22.292189 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:22.313947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-49b27636-e049-4511-9b85-6ef6f519b960 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:22.315796 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-49b27636-e049-4511-9b85-6ef6f519b960 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:22.408545 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 705/1419] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 30 09:39:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:39:22.559078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80e2db3-1234-4983-b356-42c5337de103 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:39:22.560608 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d80e2db3-1234-4983-b356-42c5337de103 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:22.702951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80e2db3-1234-4983-b356-42c5337de103 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:39:22.715008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:22.816808 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacd1121-bef4-43f7-8490-f03d1d5882a4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:39:22.817341 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacd1121-bef4-43f7-8490-f03d1d5882a4 service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:22.817620 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacd1121-bef4-43f7-8490-f03d1d5882a4 service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:22.817891 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacd1121-bef4-43f7-8490-f03d1d5882a4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:22.819535 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-bacd1121-bef4-43f7-8490-f03d1d5882a4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:22.907065 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacd1121-bef4-43f7-8490-f03d1d5882a4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:39:22.909313 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacd1121-bef4-43f7-8490-f03d1d5882a4 service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:39:22.926924 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 715/1420] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:39:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:39:24.028499 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa2a955-6457-44ba-89ab-f29ef8552d58 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:24.029027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa2a955-6457-44ba-89ab-f29ef8552d58 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:24.029027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa2a955-6457-44ba-89ab-f29ef8552d58 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:24.030329 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 706/1421] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 30 09:39:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:24.036996 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1174338-2ee9-4730-831f-5dfd2a1cb841 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:24.037444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1174338-2ee9-4730-831f-5dfd2a1cb841 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:24.037557 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1174338-2ee9-4730-831f-5dfd2a1cb841 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:24.057423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:24.112746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-b1174338-2ee9-4730-831f-5dfd2a1cb841 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:24.113960 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b1174338-2ee9-4730-831f-5dfd2a1cb841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:24.210031 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 716/1422] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 30 09:39:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:39:24.359785 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa2a955-6457-44ba-89ab-f29ef8552d58 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:39:24.361871 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-baa2a955-6457-44ba-89ab-f29ef8552d58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:24.468419 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:24.503276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa2a955-6457-44ba-89ab-f29ef8552d58 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:39:24.612863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f82323b-2079-498c-9b83-e7cb0f1f4d32 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:39:24.613389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f82323b-2079-498c-9b83-e7cb0f1f4d32 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:24.613664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f82323b-2079-498c-9b83-e7cb0f1f4d32 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:24.613933 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f82323b-2079-498c-9b83-e7cb0f1f4d32 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:24.615534 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4f82323b-2079-498c-9b83-e7cb0f1f4d32 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:24.690063 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:24.702801 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f82323b-2079-498c-9b83-e7cb0f1f4d32 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:39:24.704389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f82323b-2079-498c-9b83-e7cb0f1f4d32 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:39:24.722827 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 707/1423] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:39:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:39:25.824216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06054a4-8f42-4090-a984-d94585b5aff0 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:25.824750 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06054a4-8f42-4090-a984-d94585b5aff0 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:25.824750 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06054a4-8f42-4090-a984-d94585b5aff0 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:25.826081 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 717/1424] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 30 09:39:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:25.832315 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916213f5-c31a-4c7d-9715-1d76d62728cc None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:25.832777 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916213f5-c31a-4c7d-9715-1d76d62728cc None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:25.832878 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916213f5-c31a-4c7d-9715-1d76d62728cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:25.908892 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-916213f5-c31a-4c7d-9715-1d76d62728cc None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:25.910092 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-916213f5-c31a-4c7d-9715-1d76d62728cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:26.002881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 708/1425] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 30 09:39:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:39:26.154106 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06054a4-8f42-4090-a984-d94585b5aff0 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:39:26.155649 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d06054a4-8f42-4090-a984-d94585b5aff0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:26.239332 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:26.300708 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06054a4-8f42-4090-a984-d94585b5aff0 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:39:26.410968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009ec14c-81d2-4f01-8b57-7642400acc14 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:39:26.411563 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009ec14c-81d2-4f01-8b57-7642400acc14 service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:26.411841 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009ec14c-81d2-4f01-8b57-7642400acc14 service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:26.412108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009ec14c-81d2-4f01-8b57-7642400acc14 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:26.413681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-009ec14c-81d2-4f01-8b57-7642400acc14 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:26.461177 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:26.503973 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009ec14c-81d2-4f01-8b57-7642400acc14 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:39:26.505932 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009ec14c-81d2-4f01-8b57-7642400acc14 service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:39:26.524598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 718/1426] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:39:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:39:27.653437 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2879dc-e769-42d4-aaf2-b3cad14a3025 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:27.653981 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2879dc-e769-42d4-aaf2-b3cad14a3025 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:27.653981 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2879dc-e769-42d4-aaf2-b3cad14a3025 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:27.659552 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 709/1427] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 30 09:39:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:27.662037 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c32972-b72b-4af2-9799-a5494fb2f7ad None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:27.662434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c32972-b72b-4af2-9799-a5494fb2f7ad None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:27.662639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c32972-b72b-4af2-9799-a5494fb2f7ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:27.738484 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-a4c32972-b72b-4af2-9799-a5494fb2f7ad None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:27.739716 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a4c32972-b72b-4af2-9799-a5494fb2f7ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:27.834896 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 719/1428] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 30 09:39:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:39:27.984210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2879dc-e769-42d4-aaf2-b3cad14a3025 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:39:27.985751 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3c2879dc-e769-42d4-aaf2-b3cad14a3025 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:28.035095 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:28.130707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2879dc-e769-42d4-aaf2-b3cad14a3025 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:39:28.241684 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44d39e7-476a-49b2-b022-546d0cce718a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:39:28.242211 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44d39e7-476a-49b2-b022-546d0cce718a service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:28.242499 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44d39e7-476a-49b2-b022-546d0cce718a service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:28.242801 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44d39e7-476a-49b2-b022-546d0cce718a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:28.244396 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f44d39e7-476a-49b2-b022-546d0cce718a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:28.251836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:28.334321 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44d39e7-476a-49b2-b022-546d0cce718a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:39:28.335965 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44d39e7-476a-49b2-b022-546d0cce718a service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:39:28.353324 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 710/1429] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:39:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:39:29.452001 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267ff782-d316-4fe4-b78c-95a68659afab None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:29.452542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267ff782-d316-4fe4-b78c-95a68659afab None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:29.452542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267ff782-d316-4fe4-b78c-95a68659afab None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:29.453798 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 720/1430] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 30 09:39:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:29.460392 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7475f646-1c6f-45e9-bf2a-0f227ba57523 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:29.460851 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7475f646-1c6f-45e9-bf2a-0f227ba57523 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:29.460967 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7475f646-1c6f-45e9-bf2a-0f227ba57523 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:29.534212 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7475f646-1c6f-45e9-bf2a-0f227ba57523 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:29.535457 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7475f646-1c6f-45e9-bf2a-0f227ba57523 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:29.600772 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:29.628587 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 711/1431] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 30 09:39:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:39:29.777053 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267ff782-d316-4fe4-b78c-95a68659afab None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:39:29.778574 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-267ff782-d316-4fe4-b78c-95a68659afab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:29.796125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:29.922176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267ff782-d316-4fe4-b78c-95a68659afab None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:39:30.007754 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:30.032199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6e3f0c-b637-4687-a9f2-c6b90ce80d79 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:39:30.033251 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6e3f0c-b637-4687-a9f2-c6b90ce80d79 service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:30.033519 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6e3f0c-b637-4687-a9f2-c6b90ce80d79 service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:30.033789 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6e3f0c-b637-4687-a9f2-c6b90ce80d79 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:30.035426 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-5d6e3f0c-b637-4687-a9f2-c6b90ce80d79 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:30.127489 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6e3f0c-b637-4687-a9f2-c6b90ce80d79 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:39:30.129084 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6e3f0c-b637-4687-a9f2-c6b90ce80d79 service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:39:30.146783 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 721/1432] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:39:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:39:31.245101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ad5941-b449-4c1d-9ed4-bf5b9a8576f9 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:31.245541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ad5941-b449-4c1d-9ed4-bf5b9a8576f9 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:31.245638 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ad5941-b449-4c1d-9ed4-bf5b9a8576f9 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:31.247014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 712/1433] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 30 09:39:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:31.253701 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56cbb13-c8b8-4670-8d05-451adaca8a32 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:31.254141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56cbb13-c8b8-4670-8d05-451adaca8a32 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:31.254248 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56cbb13-c8b8-4670-8d05-451adaca8a32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:31.330899 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-f56cbb13-c8b8-4670-8d05-451adaca8a32 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:31.332108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-f56cbb13-c8b8-4670-8d05-451adaca8a32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:31.372473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:31.426248 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 722/1434] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 30 09:39:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:39:31.574968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ad5941-b449-4c1d-9ed4-bf5b9a8576f9 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:39:31.576507 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-47ad5941-b449-4c1d-9ed4-bf5b9a8576f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:31.719142 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ad5941-b449-4c1d-9ed4-bf5b9a8576f9 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:39:31.773244 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:31.830770 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927f310c-75b2-4b17-bbc7-e871e829c066 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:39:31.831310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927f310c-75b2-4b17-bbc7-e871e829c066 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:31.831593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927f310c-75b2-4b17-bbc7-e871e829c066 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:31.831868 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927f310c-75b2-4b17-bbc7-e871e829c066 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:31.833469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-927f310c-75b2-4b17-bbc7-e871e829c066 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:31.923748 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927f310c-75b2-4b17-bbc7-e871e829c066 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:39:31.925345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927f310c-75b2-4b17-bbc7-e871e829c066 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:39:31.943103 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 713/1435] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:39:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:39:33.021334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1306051-f254-4c33-9452-abb341caab2f None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:33.021898 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1306051-f254-4c33-9452-abb341caab2f None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:33.021898 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1306051-f254-4c33-9452-abb341caab2f None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:33.023276 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 723/1436] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 30 09:39:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:33.029269 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef66791a-2664-4bc8-9cc2-a684a7461987 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:33.029724 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef66791a-2664-4bc8-9cc2-a684a7461987 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:33.029827 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef66791a-2664-4bc8-9cc2-a684a7461987 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:33.104635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ef66791a-2664-4bc8-9cc2-a684a7461987 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:33.105840 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ef66791a-2664-4bc8-9cc2-a684a7461987 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:33.107842 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:33.201170 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 714/1437] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 30 09:39:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:39:33.350457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1306051-f254-4c33-9452-abb341caab2f None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:39:33.351923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c1306051-f254-4c33-9452-abb341caab2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:33.493401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1306051-f254-4c33-9452-abb341caab2f None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:39:33.520272 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:33.605706 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8c78d1-84ba-4dc5-84c3-bf23d0bad562 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:39:33.606234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8c78d1-84ba-4dc5-84c3-bf23d0bad562 service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:33.606509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8c78d1-84ba-4dc5-84c3-bf23d0bad562 service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:33.606824 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8c78d1-84ba-4dc5-84c3-bf23d0bad562 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:33.608416 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-8a8c78d1-84ba-4dc5-84c3-bf23d0bad562 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:33.694999 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8c78d1-84ba-4dc5-84c3-bf23d0bad562 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:39:33.696587 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8c78d1-84ba-4dc5-84c3-bf23d0bad562 service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:39:33.714465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 724/1438] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:39:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:39:35.772048 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb6af51-05ac-4b72-a745-4778b0f207fc None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:35.772615 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb6af51-05ac-4b72-a745-4778b0f207fc None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:35.772615 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb6af51-05ac-4b72-a745-4778b0f207fc None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:35.773993 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 715/1439] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 30 09:39:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:35.779346 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d7700e-576e-4a63-a1e0-146d91b74852 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:35.779795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d7700e-576e-4a63-a1e0-146d91b74852 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:35.779905 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d7700e-576e-4a63-a1e0-146d91b74852 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:35.821652 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:35.845927 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-97d7700e-576e-4a63-a1e0-146d91b74852 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:35.847206 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-97d7700e-576e-4a63-a1e0-146d91b74852 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:35.943304 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 725/1440] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 30 09:39:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:39:36.230949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb6af51-05ac-4b72-a745-4778b0f207fc None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:39:36.232512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-cdb6af51-05ac-4b72-a745-4778b0f207fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:36.335465 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb6af51-05ac-4b72-a745-4778b0f207fc None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:39:36.367399 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:36.407788 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0912c35e-36eb-49c9-ba06-dd561a0d7211 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:39:36.408385 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0912c35e-36eb-49c9-ba06-dd561a0d7211 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:36.408662 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0912c35e-36eb-49c9-ba06-dd561a0d7211 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:36.408929 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0912c35e-36eb-49c9-ba06-dd561a0d7211 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:36.410493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0912c35e-36eb-49c9-ba06-dd561a0d7211 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:36.517919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0912c35e-36eb-49c9-ba06-dd561a0d7211 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:39:36.518859 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0912c35e-36eb-49c9-ba06-dd561a0d7211 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:39:36.537263 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 716/1441] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:39:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:39:38.821133 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ffd950-cf0e-4a94-ac05-2bfb8018fb5d None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:38.821705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ffd950-cf0e-4a94-ac05-2bfb8018fb5d None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:38.821705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ffd950-cf0e-4a94-ac05-2bfb8018fb5d None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:38.823058 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 726/1442] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 30 09:39:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:38.828188 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b6dc6d-f6dc-4b56-a2bc-8fc9cce0b960 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:38.828632 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b6dc6d-f6dc-4b56-a2bc-8fc9cce0b960 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:38.828742 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b6dc6d-f6dc-4b56-a2bc-8fc9cce0b960 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:38.893250 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d8b6dc6d-f6dc-4b56-a2bc-8fc9cce0b960 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:38.894446 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d8b6dc6d-f6dc-4b56-a2bc-8fc9cce0b960 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:38.906898 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:38.989156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 717/1443] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 30 09:39:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:39:39.271384 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ffd950-cf0e-4a94-ac05-2bfb8018fb5d None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:39:39.272958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b8ffd950-cf0e-4a94-ac05-2bfb8018fb5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:39.374064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ffd950-cf0e-4a94-ac05-2bfb8018fb5d None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:39:39.387097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:39.447677 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef254453-2dcb-44b0-b48d-490d1e40e3cf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:39:39.448207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef254453-2dcb-44b0-b48d-490d1e40e3cf service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:39.448486 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef254453-2dcb-44b0-b48d-490d1e40e3cf service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:39.448756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef254453-2dcb-44b0-b48d-490d1e40e3cf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:39.450342 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ef254453-2dcb-44b0-b48d-490d1e40e3cf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:39.557709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef254453-2dcb-44b0-b48d-490d1e40e3cf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:39:39.559181 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef254453-2dcb-44b0-b48d-490d1e40e3cf service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:39:39.577065 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 727/1444] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:39:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:39:42.075859 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223a50b1-ceae-4e4e-a4f8-44807ca959c5 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:42.076432 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223a50b1-ceae-4e4e-a4f8-44807ca959c5 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:42.076432 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223a50b1-ceae-4e4e-a4f8-44807ca959c5 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:42.077744 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 718/1445] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 30 09:39:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:42.083137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab77c68-a9b4-46fe-8a6d-39b4d64b3920 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:42.083562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab77c68-a9b4-46fe-8a6d-39b4d64b3920 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:42.083684 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab77c68-a9b4-46fe-8a6d-39b4d64b3920 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:42.132366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:42.149961 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-7ab77c68-a9b4-46fe-8a6d-39b4d64b3920 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:42.151241 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7ab77c68-a9b4-46fe-8a6d-39b4d64b3920 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:42.247407 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 728/1446] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 30 09:39:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:39:42.527154 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223a50b1-ceae-4e4e-a4f8-44807ca959c5 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:39:42.528710 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-223a50b1-ceae-4e4e-a4f8-44807ca959c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:42.630925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223a50b1-ceae-4e4e-a4f8-44807ca959c5 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:39:42.671362 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:42.704237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04de404-97a8-4535-a154-793b9d10a7a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:39:42.704845 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04de404-97a8-4535-a154-793b9d10a7a5 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:42.705205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04de404-97a8-4535-a154-793b9d10a7a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:42.705473 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04de404-97a8-4535-a154-793b9d10a7a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:42.707824 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f04de404-97a8-4535-a154-793b9d10a7a5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:42.814392 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04de404-97a8-4535-a154-793b9d10a7a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:39:42.815777 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04de404-97a8-4535-a154-793b9d10a7a5 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:39:42.835407 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 719/1447] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:39:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:39:45.073399 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21df9b8-ebbd-4488-a2d2-96d0da933d0e None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:45.073994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21df9b8-ebbd-4488-a2d2-96d0da933d0e None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:45.073994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21df9b8-ebbd-4488-a2d2-96d0da933d0e None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:45.075325 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 729/1448] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 30 09:39:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:45.080508 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cb077d-2f97-452f-be81-7d4b90e7c20c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:45.080916 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cb077d-2f97-452f-be81-7d4b90e7c20c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:45.081045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cb077d-2f97-452f-be81-7d4b90e7c20c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:45.145427 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c9cb077d-2f97-452f-be81-7d4b90e7c20c None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:45.146640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c9cb077d-2f97-452f-be81-7d4b90e7c20c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:45.166991 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:45.243608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 720/1449] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 30 09:39:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:39:45.523693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21df9b8-ebbd-4488-a2d2-96d0da933d0e None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:39:45.525223 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a21df9b8-ebbd-4488-a2d2-96d0da933d0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:45.626732 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21df9b8-ebbd-4488-a2d2-96d0da933d0e None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:39:45.647189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:45.699635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2a78cc-35e7-478a-9be0-c2c16eb7e3b5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:39:45.700240 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2a78cc-35e7-478a-9be0-c2c16eb7e3b5 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:45.700540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2a78cc-35e7-478a-9be0-c2c16eb7e3b5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:45.700802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2a78cc-35e7-478a-9be0-c2c16eb7e3b5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:45.702397 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-3d2a78cc-35e7-478a-9be0-c2c16eb7e3b5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:45.808443 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2a78cc-35e7-478a-9be0-c2c16eb7e3b5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:39:45.809798 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2a78cc-35e7-478a-9be0-c2c16eb7e3b5 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:39:45.827881 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 730/1450] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:39:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:39:48.123102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11266a4f-f4f2-4c70-9e6f-a21662dbe0d2 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:48.123677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11266a4f-f4f2-4c70-9e6f-a21662dbe0d2 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:48.123677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11266a4f-f4f2-4c70-9e6f-a21662dbe0d2 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:48.125521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 721/1451] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 30 09:39:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:48.131151 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee43705-5630-4b1a-83a2-702ad6ac3914 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:48.131586 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee43705-5630-4b1a-83a2-702ad6ac3914 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:48.131702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee43705-5630-4b1a-83a2-702ad6ac3914 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:48.195532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:48.198524 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-bee43705-5630-4b1a-83a2-702ad6ac3914 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:48.199758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-bee43705-5630-4b1a-83a2-702ad6ac3914 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:48.295240 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 731/1452] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 30 09:39:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:39:48.579753 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11266a4f-f4f2-4c70-9e6f-a21662dbe0d2 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:39:48.581301 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-11266a4f-f4f2-4c70-9e6f-a21662dbe0d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:48.683755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11266a4f-f4f2-4c70-9e6f-a21662dbe0d2 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:39:48.732246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:48.758062 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40eda931-6cb7-45b8-be2b-b79b6247af9f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:39:48.758700 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40eda931-6cb7-45b8-be2b-b79b6247af9f service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:48.758977 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40eda931-6cb7-45b8-be2b-b79b6247af9f service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:48.759247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40eda931-6cb7-45b8-be2b-b79b6247af9f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:48.760829 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-40eda931-6cb7-45b8-be2b-b79b6247af9f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:48.867908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40eda931-6cb7-45b8-be2b-b79b6247af9f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:39:48.869253 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40eda931-6cb7-45b8-be2b-b79b6247af9f service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:39:48.888174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 722/1453] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:39:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:39:51.075369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e49e59-c3ff-4baf-bd47-04ec1a8d5897 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:51.076013 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e49e59-c3ff-4baf-bd47-04ec1a8d5897 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:51.076013 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e49e59-c3ff-4baf-bd47-04ec1a8d5897 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:51.077338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 732/1454] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 30 09:39:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:51.083198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2930d32-8258-4f98-9b1e-a84de2da5ed8 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:51.083630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2930d32-8258-4f98-9b1e-a84de2da5ed8 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:51.083736 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2930d32-8258-4f98-9b1e-a84de2da5ed8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:51.147949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f2930d32-8258-4f98-9b1e-a84de2da5ed8 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:51.149140 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f2930d32-8258-4f98-9b1e-a84de2da5ed8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:51.178029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:51.245225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 723/1455] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 30 09:39:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:39:51.536437 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e49e59-c3ff-4baf-bd47-04ec1a8d5897 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:39:51.552480 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-f1e49e59-c3ff-4baf-bd47-04ec1a8d5897 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:51.639247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e49e59-c3ff-4baf-bd47-04ec1a8d5897 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:39:51.666946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:51.712968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208908ef-cffe-4f8d-8821-4f80a301874a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:39:51.713573 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208908ef-cffe-4f8d-8821-4f80a301874a service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:51.713847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208908ef-cffe-4f8d-8821-4f80a301874a service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:51.714117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208908ef-cffe-4f8d-8821-4f80a301874a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:51.715733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-208908ef-cffe-4f8d-8821-4f80a301874a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:51.821991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208908ef-cffe-4f8d-8821-4f80a301874a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:39:51.823387 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208908ef-cffe-4f8d-8821-4f80a301874a service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:39:51.840673 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 733/1456] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:39:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:39:54.023661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42636853-1c94-4df8-ba50-9c8e119cfaab None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:54.024244 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42636853-1c94-4df8-ba50-9c8e119cfaab None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:54.024244 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42636853-1c94-4df8-ba50-9c8e119cfaab None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:54.025538 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 724/1457] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 30 09:39:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:54.030738 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226db71e-c35f-47cb-8a8f-4c0ebbe3d839 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:54.031734 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226db71e-c35f-47cb-8a8f-4c0ebbe3d839 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:54.031839 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226db71e-c35f-47cb-8a8f-4c0ebbe3d839 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:54.097596 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-226db71e-c35f-47cb-8a8f-4c0ebbe3d839 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:54.098824 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-226db71e-c35f-47cb-8a8f-4c0ebbe3d839 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:54.107034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:54.195229 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 734/1458] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 30 09:39:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:39:54.479679 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42636853-1c94-4df8-ba50-9c8e119cfaab None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:39:54.481218 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-42636853-1c94-4df8-ba50-9c8e119cfaab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:54.582746 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42636853-1c94-4df8-ba50-9c8e119cfaab None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:39:54.637099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:54.654834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9e7b51-26e2-4768-91a0-719ae1d8e809 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:39:54.655434 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9e7b51-26e2-4768-91a0-719ae1d8e809 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:54.655709 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9e7b51-26e2-4768-91a0-719ae1d8e809 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:54.655979 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9e7b51-26e2-4768-91a0-719ae1d8e809 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:54.657542 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-cb9e7b51-26e2-4768-91a0-719ae1d8e809 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:54.765200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9e7b51-26e2-4768-91a0-719ae1d8e809 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:39:54.766746 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9e7b51-26e2-4768-91a0-719ae1d8e809 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:39:54.786410 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 725/1459] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:39:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:39:57.018580 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfb32ab-dd6b-4d50-b613-81052c5f5a62 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:57.019185 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfb32ab-dd6b-4d50-b613-81052c5f5a62 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:57.019185 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfb32ab-dd6b-4d50-b613-81052c5f5a62 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:57.020523 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 735/1460] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 30 09:39:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:57.025943 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7ef722-6013-4876-b920-db317f2a04a2 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:57.026366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7ef722-6013-4876-b920-db317f2a04a2 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:57.026484 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7ef722-6013-4876-b920-db317f2a04a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:57.092041 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-1d7ef722-6013-4876-b920-db317f2a04a2 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:57.093391 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1d7ef722-6013-4876-b920-db317f2a04a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:57.130987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:57.188128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 726/1461] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 30 09:39:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:39:57.467473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfb32ab-dd6b-4d50-b613-81052c5f5a62 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:39:57.469055 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-4bfb32ab-dd6b-4d50-b613-81052c5f5a62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:57.571939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfb32ab-dd6b-4d50-b613-81052c5f5a62 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:39:57.608220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:39:57.643544 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45624ac-249e-4865-a80e-72258267fddc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:39:57.644150 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45624ac-249e-4865-a80e-72258267fddc service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:57.644413 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45624ac-249e-4865-a80e-72258267fddc service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:57.644680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45624ac-249e-4865-a80e-72258267fddc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:57.646273 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a45624ac-249e-4865-a80e-72258267fddc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:39:57.752395 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45624ac-249e-4865-a80e-72258267fddc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:39:57.753821 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45624ac-249e-4865-a80e-72258267fddc service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:39:57.771904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 736/1462] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:39:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:39:59.916597 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23ae71c-a403-4d22-b2b9-567d902f6b66 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:59.917149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23ae71c-a403-4d22-b2b9-567d902f6b66 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:59.917247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23ae71c-a403-4d22-b2b9-567d902f6b66 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:59.918527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 727/1463] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 30 09:39:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:39:59.923843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd003187-9eb5-4526-9385-95f62597d7cb None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:39:59.924268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd003187-9eb5-4526-9385-95f62597d7cb None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:39:59.924378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd003187-9eb5-4526-9385-95f62597d7cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:39:59.990912 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-bd003187-9eb5-4526-9385-95f62597d7cb None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:39:59.992156 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-bd003187-9eb5-4526-9385-95f62597d7cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:00.009661 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:40:00.086686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 737/1464] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 30 09:39:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:40:00.368602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23ae71c-a403-4d22-b2b9-567d902f6b66 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:40:00.370207 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e23ae71c-a403-4d22-b2b9-567d902f6b66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:00.470725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23ae71c-a403-4d22-b2b9-567d902f6b66 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:40:00.535194 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:40:00.541475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d294eaf-cb39-4a06-8f84-65b926be7df5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:40:00.542017 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d294eaf-cb39-4a06-8f84-65b926be7df5 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:00.542293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d294eaf-cb39-4a06-8f84-65b926be7df5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:00.542563 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d294eaf-cb39-4a06-8f84-65b926be7df5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:00.544163 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5d294eaf-cb39-4a06-8f84-65b926be7df5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:00.650987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d294eaf-cb39-4a06-8f84-65b926be7df5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:40:00.652419 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d294eaf-cb39-4a06-8f84-65b926be7df5 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:40:00.670458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 728/1465] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:40:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:40:02.820738 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532b8bdf-4104-401f-85d5-a079fbba3284 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:02.821279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532b8bdf-4104-401f-85d5-a079fbba3284 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:02.821279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532b8bdf-4104-401f-85d5-a079fbba3284 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:02.822647 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 738/1466] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 30 09:40:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:40:02.828066 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6bc705-fe65-4638-926a-d343ba811f9f None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:02.828494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6bc705-fe65-4638-926a-d343ba811f9f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:02.828650 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6bc705-fe65-4638-926a-d343ba811f9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:02.895254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-2b6bc705-fe65-4638-926a-d343ba811f9f None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:40:02.896458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2b6bc705-fe65-4638-926a-d343ba811f9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:02.943975 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:40:02.991248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 729/1467] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 30 09:40:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:40:03.270827 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532b8bdf-4104-401f-85d5-a079fbba3284 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:40:03.272372 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-532b8bdf-4104-401f-85d5-a079fbba3284 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:03.374465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532b8bdf-4104-401f-85d5-a079fbba3284 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:40:03.421844 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:40:03.446442 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29540e3e-b073-4a6f-a6b7-822aea2ef556 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:40:03.457690 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29540e3e-b073-4a6f-a6b7-822aea2ef556 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:03.457690 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29540e3e-b073-4a6f-a6b7-822aea2ef556 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:03.457690 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29540e3e-b073-4a6f-a6b7-822aea2ef556 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:03.457690 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-29540e3e-b073-4a6f-a6b7-822aea2ef556 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:03.555776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29540e3e-b073-4a6f-a6b7-822aea2ef556 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:40:03.557231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29540e3e-b073-4a6f-a6b7-822aea2ef556 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:40:03.575385 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 739/1468] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:40:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:40:12.989737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=22 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:40:13.573111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=66 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:40:23.441527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca6b05a-3951-4c50-afb5-8406312d0ade None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:23.442091 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca6b05a-3951-4c50-afb5-8406312d0ade None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:23.442091 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca6b05a-3951-4c50-afb5-8406312d0ade None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:23.443518 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 730/1469] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:40:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:40:23.448630 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bba31e-23b0-465a-9b46-2d2ddc5a38b8 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:23.449134 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bba31e-23b0-465a-9b46-2d2ddc5a38b8 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:23.449249 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bba31e-23b0-465a-9b46-2d2ddc5a38b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:23.515755 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-02bba31e-23b0-465a-9b46-2d2ddc5a38b8 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:40:23.516963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-02bba31e-23b0-465a-9b46-2d2ddc5a38b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:23.612683 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 740/1470] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:40:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:40:23.910672 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2f7b53-d096-4bf9-8b85-eba7f5043962 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:23.910958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2f7b53-d096-4bf9-8b85-eba7f5043962 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:23.911234 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2f7b53-d096-4bf9-8b85-eba7f5043962 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:23.912633 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 731/1471] 10.222.0.39 () {54 vars in 962 bytes} [Fri Sep 30 09:40:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:40:23.917491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e615eee-00da-40fc-bc6e-c5872e573413 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:23.917782 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e615eee-00da-40fc-bc6e-c5872e573413 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:23.918077 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e615eee-00da-40fc-bc6e-c5872e573413 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:23.992466 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-3e615eee-00da-40fc-bc6e-c5872e573413 None None] MFA Rules not processed for user `bf907d4033064fb8a2707b1bbb0a180b`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:40:23.993662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-3e615eee-00da-40fc-bc6e-c5872e573413 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:24.014372 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:40:24.069043 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 741/1472] 10.222.0.39 () {58 vars in 1059 bytes} [Fri Sep 30 09:40:23 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:40:24.076593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2f7b53-d096-4bf9-8b85-eba7f5043962 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:40:24.078135 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3a2f7b53-d096-4bf9-8b85-eba7f5043962 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:24.179376 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2f7b53-d096-4bf9-8b85-eba7f5043962 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:40:24.252807 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edf6d5d-e277-4282-858c-b74860a17804 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf907d4033064fb8a2707b1bbb0a180b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:40:24.253409 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edf6d5d-e277-4282-858c-b74860a17804 service glance] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:24.253690 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edf6d5d-e277-4282-858c-b74860a17804 service glance] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:24.253964 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edf6d5d-e277-4282-858c-b74860a17804 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:24.255942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5edf6d5d-e277-4282-858c-b74860a17804 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:24.310071 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:40:24.363916 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edf6d5d-e277-4282-858c-b74860a17804 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:40:24.365288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edf6d5d-e277-4282-858c-b74860a17804 service glance] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:40:24.384354 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 732/1473] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:40:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:40:26.616367 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80679f-1710-4df9-b1f5-da3cc88f9892 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:26.616971 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80679f-1710-4df9-b1f5-da3cc88f9892 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:26.617092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80679f-1710-4df9-b1f5-da3cc88f9892 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:26.618447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 742/1474] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:40:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:40:26.624350 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bd5bd3-2faa-4f98-89a0-cb9064bd39e6 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:26.624800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bd5bd3-2faa-4f98-89a0-cb9064bd39e6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:26.625045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bd5bd3-2faa-4f98-89a0-cb9064bd39e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:26.692091 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d8bd5bd3-2faa-4f98-89a0-cb9064bd39e6 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:40:26.693303 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d8bd5bd3-2faa-4f98-89a0-cb9064bd39e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:26.789279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 733/1475] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:40:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:40:27.062315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce80679f-1710-4df9-b1f5-da3cc88f9892 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:40:27.063847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ce80679f-1710-4df9-b1f5-da3cc88f9892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:27.187258 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce80679f-1710-4df9-b1f5-da3cc88f9892 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:40:27.237238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:40:27.279351 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e163b6-6ae7-4445-8ab4-2464a1417bc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:40:27.279949 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e163b6-6ae7-4445-8ab4-2464a1417bc6 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:27.280230 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e163b6-6ae7-4445-8ab4-2464a1417bc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:27.280497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e163b6-6ae7-4445-8ab4-2464a1417bc6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:27.281157 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e163b6-6ae7-4445-8ab4-2464a1417bc6 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:40:27.283733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: WARNING py.warnings [None req-38e163b6-6ae7-4445-8ab4-2464a1417bc6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 30 09:40:27.283733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: warnings.warn(msg) Sep 30 09:40:27.283733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: Sep 30 09:40:27.284868 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e163b6-6ae7-4445-8ab4-2464a1417bc6 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:40:27.303904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 743/1476] 10.222.0.39 () {56 vars in 1108 bytes} [Fri Sep 30 09:40:27 2022] GET /identity/v3/projects => generated 2553 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:40:29.460693 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1b16d0-acbb-4a82-ab89-4961ec9067c1 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:29.461230 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1b16d0-acbb-4a82-ab89-4961ec9067c1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:29.461230 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1b16d0-acbb-4a82-ab89-4961ec9067c1 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:29.462485 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 734/1477] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:40:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:40:29.467922 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c594ec98-641b-4e90-a0d7-cfc0039793cd None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:29.468408 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c594ec98-641b-4e90-a0d7-cfc0039793cd None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:29.468516 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c594ec98-641b-4e90-a0d7-cfc0039793cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:29.533293 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-c594ec98-641b-4e90-a0d7-cfc0039793cd None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:40:29.534505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c594ec98-641b-4e90-a0d7-cfc0039793cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:29.628822 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 744/1478] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:40:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:40:29.921018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1b16d0-acbb-4a82-ab89-4961ec9067c1 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:40:29.922534 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6b1b16d0-acbb-4a82-ab89-4961ec9067c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:29.932406 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:40:30.067241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1b16d0-acbb-4a82-ab89-4961ec9067c1 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:40:30.146704 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:40:30.180136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69551092-a2c6-4c31-a485-6b9e873d1894 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:40:30.180673 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69551092-a2c6-4c31-a485-6b9e873d1894 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:30.180951 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69551092-a2c6-4c31-a485-6b9e873d1894 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:30.181223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69551092-a2c6-4c31-a485-6b9e873d1894 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:30.182858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-69551092-a2c6-4c31-a485-6b9e873d1894 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:30.270992 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69551092-a2c6-4c31-a485-6b9e873d1894 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:40:30.272582 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69551092-a2c6-4c31-a485-6b9e873d1894 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:40:30.290641 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 735/1479] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:40:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:40:32.586621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e6b459-a20e-4f84-8c7f-69ca2adced79 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:32.587216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e6b459-a20e-4f84-8c7f-69ca2adced79 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:32.587216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e6b459-a20e-4f84-8c7f-69ca2adced79 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:32.588533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 745/1480] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:40:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:40:32.594343 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f688310-aa25-461b-a320-a5578bac08b8 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:32.594789 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f688310-aa25-461b-a320-a5578bac08b8 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:32.594932 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f688310-aa25-461b-a320-a5578bac08b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:32.661897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-1f688310-aa25-461b-a320-a5578bac08b8 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:40:32.663175 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1f688310-aa25-461b-a320-a5578bac08b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:32.708360 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:40:32.757619 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 736/1481] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:40:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:40:33.046222 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e6b459-a20e-4f84-8c7f-69ca2adced79 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:40:33.047812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-75e6b459-a20e-4f84-8c7f-69ca2adced79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:33.072001 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:40:33.151892 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e6b459-a20e-4f84-8c7f-69ca2adced79 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:40:33.225086 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e076a5b-0f8e-46ff-b4a2-6cf71bad9807 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:40:33.225685 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e076a5b-0f8e-46ff-b4a2-6cf71bad9807 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:33.225958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e076a5b-0f8e-46ff-b4a2-6cf71bad9807 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:33.226238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e076a5b-0f8e-46ff-b4a2-6cf71bad9807 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:33.227844 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-9e076a5b-0f8e-46ff-b4a2-6cf71bad9807 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:33.300618 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:40:33.335634 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e076a5b-0f8e-46ff-b4a2-6cf71bad9807 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:40:33.337072 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e076a5b-0f8e-46ff-b4a2-6cf71bad9807 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:40:33.354531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 746/1482] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:40:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:40:35.540499 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5141dd9e-9e0a-43cb-b396-3c14414f6b2c None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:35.541075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5141dd9e-9e0a-43cb-b396-3c14414f6b2c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:35.541075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5141dd9e-9e0a-43cb-b396-3c14414f6b2c None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:35.542418 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 737/1483] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:40:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:40:35.548200 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b4e425-fa30-4bff-9075-88d10463e9c8 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:35.548597 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b4e425-fa30-4bff-9075-88d10463e9c8 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:35.548746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b4e425-fa30-4bff-9075-88d10463e9c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:35.613352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-a7b4e425-fa30-4bff-9075-88d10463e9c8 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:40:35.614560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a7b4e425-fa30-4bff-9075-88d10463e9c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:35.711618 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 747/1484] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:40:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:40:36.001018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5141dd9e-9e0a-43cb-b396-3c14414f6b2c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:40:36.002584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5141dd9e-9e0a-43cb-b396-3c14414f6b2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:36.104710 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5141dd9e-9e0a-43cb-b396-3c14414f6b2c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:40:36.174928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:40:36.176345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c7c612-2096-43e0-9b90-402abc962bd7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:40:36.177010 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c7c612-2096-43e0-9b90-402abc962bd7 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:36.177287 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c7c612-2096-43e0-9b90-402abc962bd7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:36.177575 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c7c612-2096-43e0-9b90-402abc962bd7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:36.179669 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-39c7c612-2096-43e0-9b90-402abc962bd7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:36.287231 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c7c612-2096-43e0-9b90-402abc962bd7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:40:36.288596 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c7c612-2096-43e0-9b90-402abc962bd7 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:40:36.305942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 738/1485] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:40:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:40:38.547193 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2319d2-5341-4698-a40b-8c3690b2e14c None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:38.547748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2319d2-5341-4698-a40b-8c3690b2e14c None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:38.547748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2319d2-5341-4698-a40b-8c3690b2e14c None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:38.549070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 748/1486] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:40:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:40:38.554357 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7118cd-ba2a-43ce-96f8-9b382e2ef5d3 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:38.554795 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7118cd-ba2a-43ce-96f8-9b382e2ef5d3 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:38.554973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7118cd-ba2a-43ce-96f8-9b382e2ef5d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:38.622058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-fc7118cd-ba2a-43ce-96f8-9b382e2ef5d3 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:40:38.623364 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fc7118cd-ba2a-43ce-96f8-9b382e2ef5d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:38.675471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:40:38.719843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 739/1487] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:40:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:40:39.013321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2319d2-5341-4698-a40b-8c3690b2e14c None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:40:39.014900 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7e2319d2-5341-4698-a40b-8c3690b2e14c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:39.049885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:40:39.115215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2319d2-5341-4698-a40b-8c3690b2e14c None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:40:39.187726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c632cdd6-e8db-42f5-85fa-774883f7a0f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:40:39.188327 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c632cdd6-e8db-42f5-85fa-774883f7a0f7 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:39.188602 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c632cdd6-e8db-42f5-85fa-774883f7a0f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:39.188865 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c632cdd6-e8db-42f5-85fa-774883f7a0f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:39.190428 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c632cdd6-e8db-42f5-85fa-774883f7a0f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:39.270877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:40:39.299428 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c632cdd6-e8db-42f5-85fa-774883f7a0f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:40:39.300871 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c632cdd6-e8db-42f5-85fa-774883f7a0f7 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:40:39.319392 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 749/1488] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:40:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:40:41.800840 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00ef46a-bf68-48bb-8a4b-a200e9a9abc1 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:41.801415 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00ef46a-bf68-48bb-8a4b-a200e9a9abc1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:41.801415 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00ef46a-bf68-48bb-8a4b-a200e9a9abc1 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:41.802764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 740/1489] 10.222.0.39 () {54 vars in 927 bytes} [Fri Sep 30 09:40:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:40:41.808225 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c549044-956a-4db5-9200-62a17592e0dc None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:41.808641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c549044-956a-4db5-9200-62a17592e0dc None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:41.808773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c549044-956a-4db5-9200-62a17592e0dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:41.874297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-3c549044-956a-4db5-9200-62a17592e0dc None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:40:41.875550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-3c549044-956a-4db5-9200-62a17592e0dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:41.973970 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 750/1490] 10.222.0.39 () {58 vars in 1024 bytes} [Fri Sep 30 09:40:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:40:42.262722 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00ef46a-bf68-48bb-8a4b-a200e9a9abc1 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:40:42.264295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a00ef46a-bf68-48bb-8a4b-a200e9a9abc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:42.365934 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00ef46a-bf68-48bb-8a4b-a200e9a9abc1 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:40:42.439253 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ce0688-4b28-4e74-9ec4-649b84c9245f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:40:42.439854 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ce0688-4b28-4e74-9ec4-649b84c9245f service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:40:42.440132 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ce0688-4b28-4e74-9ec4-649b84c9245f service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:40:42.440403 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ce0688-4b28-4e74-9ec4-649b84c9245f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:40:42.441988 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-51ce0688-4b28-4e74-9ec4-649b84c9245f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:40:42.450335 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:40:42.549210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ce0688-4b28-4e74-9ec4-649b84c9245f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:40:42.550783 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ce0688-4b28-4e74-9ec4-649b84c9245f service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:40:42.568777 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 741/1491] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:40:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:40:51.972428 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=82 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:40:52.566632 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=50 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:42:24.239379 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b7f537-123e-48cd-aaa2-c7bb6b957a56 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:42:24.239904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b7f537-123e-48cd-aaa2-c7bb6b957a56 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:42:24.239904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b7f537-123e-48cd-aaa2-c7bb6b957a56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:42:24.314886 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-09b7f537-123e-48cd-aaa2-c7bb6b957a56 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:42:24.316110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-09b7f537-123e-48cd-aaa2-c7bb6b957a56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:42:24.411584 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 751/1492] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:42:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:42:24.419541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a84554-8c7d-422d-9a08-cf48c4bc5af7 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:42:24.420017 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a84554-8c7d-422d-9a08-cf48c4bc5af7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:42:24.420125 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a84554-8c7d-422d-9a08-cf48c4bc5af7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:42:24.495498 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-90a84554-8c7d-422d-9a08-cf48c4bc5af7 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:42:24.496706 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-90a84554-8c7d-422d-9a08-cf48c4bc5af7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:42:24.591687 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 742/1493] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:42:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:42:24.596267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b7f537-123e-48cd-aaa2-c7bb6b957a56 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:42:24.597488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-09b7f537-123e-48cd-aaa2-c7bb6b957a56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:42:24.608743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b7f537-123e-48cd-aaa2-c7bb6b957a56 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:42:24.670771 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:42:24.703948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198e5c69-cfc5-4374-af90-0f26e7a8315d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:42:24.704689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198e5c69-cfc5-4374-af90-0f26e7a8315d admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:42:24.704969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198e5c69-cfc5-4374-af90-0f26e7a8315d admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:42:24.705238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198e5c69-cfc5-4374-af90-0f26e7a8315d admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:42:24.705869 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198e5c69-cfc5-4374-af90-0f26e7a8315d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:42:24.706166 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198e5c69-cfc5-4374-af90-0f26e7a8315d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:42:24.707204 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: WARNING py.warnings [None req-198e5c69-cfc5-4374-af90-0f26e7a8315d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 30 09:42:24.707204 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: warnings.warn(msg) Sep 30 09:42:24.707204 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: Sep 30 09:42:24.708113 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198e5c69-cfc5-4374-af90-0f26e7a8315d admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:42:24.720013 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 752/1494] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:42:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:42:24.728383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a84554-8c7d-422d-9a08-cf48c4bc5af7 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:42:24.737475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a84554-8c7d-422d-9a08-cf48c4bc5af7 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:42:24.798837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:42:24.833058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbeb550-4943-4bcd-9398-3118430feb79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:42:24.833664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbeb550-4943-4bcd-9398-3118430feb79 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:42:24.833939 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbeb550-4943-4bcd-9398-3118430feb79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:42:24.834205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbeb550-4943-4bcd-9398-3118430feb79 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:42:24.834938 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbeb550-4943-4bcd-9398-3118430feb79 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:42:24.836808 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: WARNING py.warnings [None req-afbeb550-4943-4bcd-9398-3118430feb79 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 30 09:42:24.836808 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: warnings.warn(msg) Sep 30 09:42:24.836808 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: Sep 30 09:42:24.838047 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbeb550-4943-4bcd-9398-3118430feb79 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:42:24.947045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 743/1495] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:42:24 2022] POST /identity/v3/projects => generated 379 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:42:24.951194 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198e5c69-cfc5-4374-af90-0f26e7a8315d admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:42:24.954223 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-198e5c69-cfc5-4374-af90-0f26e7a8315d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:42:24.989407 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198e5c69-cfc5-4374-af90-0f26e7a8315d admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:42:25.087139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f068a9f-bfd9-481e-b314-9bb81c27e700 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:42:25.087660 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f068a9f-bfd9-481e-b314-9bb81c27e700 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:42:25.087935 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f068a9f-bfd9-481e-b314-9bb81c27e700 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:42:25.088204 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f068a9f-bfd9-481e-b314-9bb81c27e700 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:42:25.088827 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f068a9f-bfd9-481e-b314-9bb81c27e700 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:42:25.089837 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: WARNING py.warnings [None req-1f068a9f-bfd9-481e-b314-9bb81c27e700 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 30 09:42:25.089837 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: warnings.warn(msg) Sep 30 09:42:25.089837 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: Sep 30 09:42:25.091075 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f068a9f-bfd9-481e-b314-9bb81c27e700 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:42:25.186742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 753/1496] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:42:24 2022] POST /identity/v3/users => generated 290 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:42:25.190739 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbeb550-4943-4bcd-9398-3118430feb79 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:42:25.192284 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-afbeb550-4943-4bcd-9398-3118430feb79 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:42:25.254416 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbeb550-4943-4bcd-9398-3118430feb79 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:42:25.335670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:42:25.345443 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac41173-fff5-4587-b866-8de6f00c77d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:42:25.345981 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac41173-fff5-4587-b866-8de6f00c77d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:42:25.346256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac41173-fff5-4587-b866-8de6f00c77d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:42:25.346526 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac41173-fff5-4587-b866-8de6f00c77d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:42:25.347180 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac41173-fff5-4587-b866-8de6f00c77d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:42:25.348211 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: WARNING py.warnings [None req-7ac41173-fff5-4587-b866-8de6f00c77d1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 30 09:42:25.348211 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: warnings.warn(msg) Sep 30 09:42:25.348211 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: Sep 30 09:42:25.349091 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac41173-fff5-4587-b866-8de6f00c77d1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:42:25.362364 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 744/1497] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:42:25 2022] GET /identity/v3/roles => generated 1420 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:42:25.366432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f068a9f-bfd9-481e-b314-9bb81c27e700 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:42:25.368430 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-1f068a9f-bfd9-481e-b314-9bb81c27e700 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:42:25.381191 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f068a9f-bfd9-481e-b314-9bb81c27e700 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:42:25.385747 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717d1250-a5e6-48dd-8d78-e87f6afb44d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:42:25.386329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717d1250-a5e6-48dd-8d78-e87f6afb44d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:42:25.386636 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717d1250-a5e6-48dd-8d78-e87f6afb44d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:42:25.386909 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717d1250-a5e6-48dd-8d78-e87f6afb44d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:42:25.387536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717d1250-a5e6-48dd-8d78-e87f6afb44d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:42:25.388536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: WARNING py.warnings [None req-717d1250-a5e6-48dd-8d78-e87f6afb44d1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 30 09:42:25.388536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: warnings.warn(msg) Sep 30 09:42:25.388536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: Sep 30 09:42:25.389413 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717d1250-a5e6-48dd-8d78-e87f6afb44d1 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:42:25.402495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 754/1498] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:42:25 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:42:25.406513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac41173-fff5-4587-b866-8de6f00c77d1 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:42:25.408037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7ac41173-fff5-4587-b866-8de6f00c77d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:42:25.440443 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac41173-fff5-4587-b866-8de6f00c77d1 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:42:25.533885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80cf5d0-2574-47d2-8e73-ef8e14045980 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:42:25.534541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80cf5d0-2574-47d2-8e73-ef8e14045980 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:42:25.534845 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80cf5d0-2574-47d2-8e73-ef8e14045980 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:42:25.535114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80cf5d0-2574-47d2-8e73-ef8e14045980 admin admin] PATH_INFO: `/v3/projects/1118d289e02347d1bb08a0008d5d64d0/users/c7175bea2f0843fd969c5d78f28ed87f/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:42:25.556671 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80cf5d0-2574-47d2-8e73-ef8e14045980 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1118d289e02347d1bb08a0008d5d64d0, user_id=c7175bea2f0843fd969c5d78f28ed87f, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:42:25.557693 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: WARNING py.warnings [None req-b80cf5d0-2574-47d2-8e73-ef8e14045980 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 30 09:42:25.557693 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: warnings.warn(msg) Sep 30 09:42:25.557693 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: Sep 30 09:42:25.559668 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80cf5d0-2574-47d2-8e73-ef8e14045980 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:42:25.696047 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 745/1499] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:42:25 2022] PUT /identity/v3/projects/1118d289e02347d1bb08a0008d5d64d0/users/c7175bea2f0843fd969c5d78f28ed87f/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:42:25.708023 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daefc600-9ef2-49ce-a19a-dbafffe0fdb2 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:42:25.708318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daefc600-9ef2-49ce-a19a-dbafffe0fdb2 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:42:25.708626 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daefc600-9ef2-49ce-a19a-dbafffe0fdb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:42:25.734111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:42:25.759212 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-daefc600-9ef2-49ce-a19a-dbafffe0fdb2 None None] MFA Rules not processed for user `c7175bea2f0843fd969c5d78f28ed87f`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:42:25.762345 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-daefc600-9ef2-49ce-a19a-dbafffe0fdb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:42:25.822984 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 755/1500] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:42:25 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:42:25.836812 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c719ef9a-7ada-4984-abbd-86409a8c816e None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:42:25.837099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c719ef9a-7ada-4984-abbd-86409a8c816e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:42:25.837429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c719ef9a-7ada-4984-abbd-86409a8c816e None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:42:25.838792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 746/1501] 10.222.0.39 () {54 vars in 963 bytes} [Fri Sep 30 09:42:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:42:25.843541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de14585-771c-4fe1-89f5-bd15c6e811bc None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:42:25.843828 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de14585-771c-4fe1-89f5-bd15c6e811bc None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:42:25.844100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de14585-771c-4fe1-89f5-bd15c6e811bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:42:25.911565 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-4de14585-771c-4fe1-89f5-bd15c6e811bc None None] MFA Rules not processed for user `4e2742f567444dff88ed919be52217de`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:42:25.912769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-4de14585-771c-4fe1-89f5-bd15c6e811bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:42:25.987238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 756/1502] 10.222.0.39 () {58 vars in 1060 bytes} [Fri Sep 30 09:42:25 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:42:25.995430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c719ef9a-7ada-4984-abbd-86409a8c816e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:42:26.007408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c719ef9a-7ada-4984-abbd-86409a8c816e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:42:26.010946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988e7203-3343-48c3-83ae-b1e99c7dd6af service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:42:26.011534 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988e7203-3343-48c3-83ae-b1e99c7dd6af service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:42:26.011816 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988e7203-3343-48c3-83ae-b1e99c7dd6af service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:42:26.012089 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988e7203-3343-48c3-83ae-b1e99c7dd6af service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:42:26.020254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988e7203-3343-48c3-83ae-b1e99c7dd6af service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:42:26.021672 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988e7203-3343-48c3-83ae-b1e99c7dd6af service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:42:26.030509 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 747/1503] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:42:25 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:42:26.076023 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbb9f70-9283-40a9-98a8-04f0dcc7a142 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:42:26.076334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbb9f70-9283-40a9-98a8-04f0dcc7a142 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:42:26.076590 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbb9f70-9283-40a9-98a8-04f0dcc7a142 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:42:26.077841 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 757/1504] 10.222.0.39 () {52 vars in 830 bytes} [Fri Sep 30 09:42:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:42:26.081448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988e7203-3343-48c3-83ae-b1e99c7dd6af service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:42:26.114303 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:42:26.142362 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988e7203-3343-48c3-83ae-b1e99c7dd6af service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:42:26.147167 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b513eeb4-881a-44e1-83fa-79e28b9e8d81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:42:26.147795 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b513eeb4-881a-44e1-83fa-79e28b9e8d81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:42:26.148078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b513eeb4-881a-44e1-83fa-79e28b9e8d81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:42:26.148353 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b513eeb4-881a-44e1-83fa-79e28b9e8d81 admin admin] PATH_INFO: `/v3/users/c7175bea2f0843fd969c5d78f28ed87f` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:42:26.149477 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b513eeb4-881a-44e1-83fa-79e28b9e8d81 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c7175bea2f0843fd969c5d78f28ed87f)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:42:26.150488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: WARNING py.warnings [None req-b513eeb4-881a-44e1-83fa-79e28b9e8d81 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 30 09:42:26.150488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: warnings.warn(msg) Sep 30 09:42:26.150488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: Sep 30 09:42:26.151822 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b513eeb4-881a-44e1-83fa-79e28b9e8d81 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:42:26.367138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b513eeb4-881a-44e1-83fa-79e28b9e8d81 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c7175bea2f0843fd969c5d78f28ed87f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:42:26.367688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b513eeb4-881a-44e1-83fa-79e28b9e8d81 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c7175bea2f0843fd969c5d78f28ed87f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:42:26.368404 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b513eeb4-881a-44e1-83fa-79e28b9e8d81 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c7175bea2f0843fd969c5d78f28ed87f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:42:26.383312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b513eeb4-881a-44e1-83fa-79e28b9e8d81 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c7175bea2f0843fd969c5d78f28ed87f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:42:26.436675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b513eeb4-881a-44e1-83fa-79e28b9e8d81 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c7175bea2f0843fd969c5d78f28ed87f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:42:26.464692 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 748/1505] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:42:26 2022] DELETE /identity/v3/users/c7175bea2f0843fd969c5d78f28ed87f => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:42:26.469048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c2a5ed-9f50-4169-a909-4745d86cc669 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:42:26.469478 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c2a5ed-9f50-4169-a909-4745d86cc669 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:42:26.469596 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c2a5ed-9f50-4169-a909-4745d86cc669 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:42:26.498276 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-b4c2a5ed-9f50-4169-a909-4745d86cc669 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:42:26.499549 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b4c2a5ed-9f50-4169-a909-4745d86cc669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:42:26.518049 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:42:26.559877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 758/1506] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:42:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:42:26.588732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17abfc6-8e11-49fa-89f2-824c12dda00f None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:42:26.588732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17abfc6-8e11-49fa-89f2-824c12dda00f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:42:26.588732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17abfc6-8e11-49fa-89f2-824c12dda00f None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:42:26.588732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 749/1507] 10.222.0.39 () {54 vars in 964 bytes} [Fri Sep 30 09:42:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:42:26.591665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bede69ac-2bfa-4ee5-9b90-4404d7b3dbeb None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:42:26.591980 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bede69ac-2bfa-4ee5-9b90-4404d7b3dbeb None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:42:26.592246 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bede69ac-2bfa-4ee5-9b90-4404d7b3dbeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:42:26.642083 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-bede69ac-2bfa-4ee5-9b90-4404d7b3dbeb None None] MFA Rules not processed for user `9266babd21414ae9998716e5a0478e9d`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:42:26.643258 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-bede69ac-2bfa-4ee5-9b90-4404d7b3dbeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:42:26.703569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 759/1508] 10.222.0.39 () {58 vars in 1061 bytes} [Fri Sep 30 09:42:26 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:42:26.713436 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17abfc6-8e11-49fa-89f2-824c12dda00f None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:42:26.724277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17abfc6-8e11-49fa-89f2-824c12dda00f None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:42:26.728394 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2445b2d-3301-48fe-9690-08c853305a40 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:42:26.728971 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2445b2d-3301-48fe-9690-08c853305a40 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:42:26.729250 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2445b2d-3301-48fe-9690-08c853305a40 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:42:26.729550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2445b2d-3301-48fe-9690-08c853305a40 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:42:26.738853 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2445b2d-3301-48fe-9690-08c853305a40 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:42:26.743973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2445b2d-3301-48fe-9690-08c853305a40 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:42:26.749838 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 750/1509] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:42:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:42:27.587615 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bede69ac-2bfa-4ee5-9b90-4404d7b3dbeb None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:42:27.589193 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-bede69ac-2bfa-4ee5-9b90-4404d7b3dbeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:42:27.603186 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bede69ac-2bfa-4ee5-9b90-4404d7b3dbeb None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:42:27.608952 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7d8ecc-34ab-49f9-97bd-e752bf1187b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:42:27.609723 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7d8ecc-34ab-49f9-97bd-e752bf1187b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:42:27.610010 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7d8ecc-34ab-49f9-97bd-e752bf1187b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:42:27.610281 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7d8ecc-34ab-49f9-97bd-e752bf1187b2 admin admin] PATH_INFO: `/v3/projects/1118d289e02347d1bb08a0008d5d64d0` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:42:27.611905 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7d8ecc-34ab-49f9-97bd-e752bf1187b2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1118d289e02347d1bb08a0008d5d64d0)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:42:27.612932 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: WARNING py.warnings [None req-da7d8ecc-34ab-49f9-97bd-e752bf1187b2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 30 09:42:27.612932 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: warnings.warn(msg) Sep 30 09:42:27.612932 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: Sep 30 09:42:27.614117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7d8ecc-34ab-49f9-97bd-e752bf1187b2 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:42:27.723064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-da7d8ecc-34ab-49f9-97bd-e752bf1187b2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1118d289e02347d1bb08a0008d5d64d0', 'request_id': 'req-da7d8ecc-34ab-49f9-97bd-e752bf1187b2'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:42:27.729172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-da7d8ecc-34ab-49f9-97bd-e752bf1187b2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1118d289e02347d1bb08a0008d5d64d0', 'request_id': 'req-da7d8ecc-34ab-49f9-97bd-e752bf1187b2'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:42:27.742766 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-da7d8ecc-34ab-49f9-97bd-e752bf1187b2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1118d289e02347d1bb08a0008d5d64d0', 'request_id': 'req-da7d8ecc-34ab-49f9-97bd-e752bf1187b2'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:42:27.808090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-da7d8ecc-34ab-49f9-97bd-e752bf1187b2 admin admin] The token cache is being invalidate because project 1118d289e02347d1bb08a0008d5d64d0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:42:27.808381 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-da7d8ecc-34ab-49f9-97bd-e752bf1187b2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:42:27.809086 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-da7d8ecc-34ab-49f9-97bd-e752bf1187b2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:42:27.810623 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 760/1510] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:42:27 2022] DELETE /identity/v3/projects/1118d289e02347d1bb08a0008d5d64d0 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:42:36.745561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=44,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:42:37.804996 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=81,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:43:54.374300 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadacd0a-e650-4ac6-9914-c528f31b7a47 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:43:54.374940 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadacd0a-e650-4ac6-9914-c528f31b7a47 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:43:54.374940 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadacd0a-e650-4ac6-9914-c528f31b7a47 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:43:54.376584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 751/1511] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 30 09:43:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:43:54.381414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18315a3-82b7-45bd-a22f-4e5da13c4daa None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:43:54.381839 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18315a3-82b7-45bd-a22f-4e5da13c4daa None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:43:54.381940 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18315a3-82b7-45bd-a22f-4e5da13c4daa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:43:54.421332 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-f18315a3-82b7-45bd-a22f-4e5da13c4daa None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:43:54.422555 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-f18315a3-82b7-45bd-a22f-4e5da13c4daa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:43:54.479875 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 761/1512] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 30 09:43:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:43:54.635160 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eadacd0a-e650-4ac6-9914-c528f31b7a47 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:43:54.648100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-eadacd0a-e650-4ac6-9914-c528f31b7a47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:43:54.737981 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eadacd0a-e650-4ac6-9914-c528f31b7a47 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:43:54.743353 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c377fa62-9ab8-4803-94ed-535e4b097a17 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:43:54.743954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c377fa62-9ab8-4803-94ed-535e4b097a17 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:43:54.744232 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c377fa62-9ab8-4803-94ed-535e4b097a17 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:43:54.744503 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c377fa62-9ab8-4803-94ed-535e4b097a17 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:43:54.752695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c377fa62-9ab8-4803-94ed-535e4b097a17 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:43:54.754846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c377fa62-9ab8-4803-94ed-535e4b097a17 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:43:54.764071 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 752/1513] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:43:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:43:55.835212 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7cb66e-50b6-42f3-a5e1-3e8653578ce9 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:43:55.835784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7cb66e-50b6-42f3-a5e1-3e8653578ce9 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:43:55.835784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7cb66e-50b6-42f3-a5e1-3e8653578ce9 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:43:55.837088 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 762/1514] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 30 09:43:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:43:55.843137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f902dab-68b9-432c-91f5-1dab04b9d946 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:43:55.843596 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f902dab-68b9-432c-91f5-1dab04b9d946 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:43:55.843701 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f902dab-68b9-432c-91f5-1dab04b9d946 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:43:55.863299 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7f902dab-68b9-432c-91f5-1dab04b9d946 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:43:55.864481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7f902dab-68b9-432c-91f5-1dab04b9d946 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:43:55.872136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 753/1515] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 30 09:43:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:43:56.022523 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7cb66e-50b6-42f3-a5e1-3e8653578ce9 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:43:56.037470 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7cb66e-50b6-42f3-a5e1-3e8653578ce9 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:43:56.043032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9e3891-ad6b-48e3-ab60-b7b4bd08b170 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:43:56.043621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9e3891-ad6b-48e3-ab60-b7b4bd08b170 service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:43:56.043909 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9e3891-ad6b-48e3-ab60-b7b4bd08b170 service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:43:56.044188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9e3891-ad6b-48e3-ab60-b7b4bd08b170 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:43:56.052760 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9e3891-ad6b-48e3-ab60-b7b4bd08b170 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:43:56.054352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9e3891-ad6b-48e3-ab60-b7b4bd08b170 service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:43:56.063742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 763/1516] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:43:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:43:56.376216 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f080d8-9e04-4387-aae7-f44ae54e3ea7 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:43:56.376432 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f080d8-9e04-4387-aae7-f44ae54e3ea7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:43:56.376709 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f080d8-9e04-4387-aae7-f44ae54e3ea7 None None] PATH_INFO: `/v3` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:43:56.377990 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 754/1517] 10.222.0.36 () {54 vars in 925 bytes} [Fri Sep 30 09:43:56 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:43:56.384045 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9790e16-f638-4f9b-81d1-912476be8fe6 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:43:56.384335 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9790e16-f638-4f9b-81d1-912476be8fe6 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:43:56.384608 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9790e16-f638-4f9b-81d1-912476be8fe6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:43:56.435272 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-e9790e16-f638-4f9b-81d1-912476be8fe6 None None] MFA Rules not processed for user `49a3011c103b4e48904f876f81d27fba`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:43:56.436473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-e9790e16-f638-4f9b-81d1-912476be8fe6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:43:56.507605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 764/1518] 10.222.0.36 () {58 vars in 1015 bytes} [Fri Sep 30 09:43:56 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:44:05.271712 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93833a67-3146-4b8f-9bbb-49070113ea9a None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:05.275917 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93833a67-3146-4b8f-9bbb-49070113ea9a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:05.275917 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93833a67-3146-4b8f-9bbb-49070113ea9a None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:05.275917 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 755/1519] 10.222.0.39 () {54 vars in 961 bytes} [Fri Sep 30 09:44:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:44:05.278579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98987748-b5a6-4480-945e-d18c1f26bae4 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:05.279119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98987748-b5a6-4480-945e-d18c1f26bae4 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:05.279221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98987748-b5a6-4480-945e-d18c1f26bae4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:05.298820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-98987748-b5a6-4480-945e-d18c1f26bae4 None None] MFA Rules not processed for user `49a3011c103b4e48904f876f81d27fba`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:44:05.300072 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-98987748-b5a6-4480-945e-d18c1f26bae4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:44:05.308725 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 765/1520] 10.222.0.39 () {58 vars in 1058 bytes} [Fri Sep 30 09:44:05 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:44:05.316887 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93833a67-3146-4b8f-9bbb-49070113ea9a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:44:05.330185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93833a67-3146-4b8f-9bbb-49070113ea9a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:44:05.335712 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd981ad-640b-493f-8f13-a8906e67f786 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '49a3011c103b4e48904f876f81d27fba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:44:05.336286 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd981ad-640b-493f-8f13-a8906e67f786 service placement] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:05.336562 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd981ad-640b-493f-8f13-a8906e67f786 service placement] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:05.336829 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd981ad-640b-493f-8f13-a8906e67f786 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:05.344742 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd981ad-640b-493f-8f13-a8906e67f786 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:44:05.346323 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd981ad-640b-493f-8f13-a8906e67f786 service placement] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:44:05.355929 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 756/1521] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:44:05 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:44:07.534567 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6612f8d-d6df-49d8-8825-fbb528d7c242 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:07.535159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6612f8d-d6df-49d8-8825-fbb528d7c242 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:07.535159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6612f8d-d6df-49d8-8825-fbb528d7c242 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:07.536467 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 766/1522] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 30 09:44:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:44:07.542582 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bbd911-ee4e-4d91-9b75-f2f371662e54 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:07.543010 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bbd911-ee4e-4d91-9b75-f2f371662e54 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:07.543147 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bbd911-ee4e-4d91-9b75-f2f371662e54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:07.564400 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-79bbd911-ee4e-4d91-9b75-f2f371662e54 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:44:07.565612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-79bbd911-ee4e-4d91-9b75-f2f371662e54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:44:07.573186 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 757/1523] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 30 09:44:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:44:07.723882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6612f8d-d6df-49d8-8825-fbb528d7c242 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:44:07.738994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6612f8d-d6df-49d8-8825-fbb528d7c242 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:44:07.744845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af7d9b7-b4ee-4b20-abab-90d86b39a41c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:44:07.749975 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af7d9b7-b4ee-4b20-abab-90d86b39a41c service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:07.749975 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af7d9b7-b4ee-4b20-abab-90d86b39a41c service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:07.749975 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af7d9b7-b4ee-4b20-abab-90d86b39a41c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:07.754273 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af7d9b7-b4ee-4b20-abab-90d86b39a41c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:44:07.755897 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af7d9b7-b4ee-4b20-abab-90d86b39a41c service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:44:07.765084 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 767/1524] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:44:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:44:08.831370 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf24294-880d-40d2-b1a7-15785f1800f5 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:08.831903 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf24294-880d-40d2-b1a7-15785f1800f5 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:08.831903 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf24294-880d-40d2-b1a7-15785f1800f5 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:08.833225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 758/1525] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 30 09:44:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:44:08.839167 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d29a99-6223-4a08-ad17-b8f9cc5af73f None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:08.839582 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d29a99-6223-4a08-ad17-b8f9cc5af73f None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:08.839717 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d29a99-6223-4a08-ad17-b8f9cc5af73f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:08.859248 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-97d29a99-6223-4a08-ad17-b8f9cc5af73f None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:44:08.860438 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-97d29a99-6223-4a08-ad17-b8f9cc5af73f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:44:08.868226 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 768/1526] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 30 09:44:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:44:09.017235 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf24294-880d-40d2-b1a7-15785f1800f5 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:44:09.032029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf24294-880d-40d2-b1a7-15785f1800f5 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:44:09.037429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f0cc8e-a551-4f4b-bbc8-dc5c75cc62b7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:44:09.038017 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f0cc8e-a551-4f4b-bbc8-dc5c75cc62b7 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:09.038296 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f0cc8e-a551-4f4b-bbc8-dc5c75cc62b7 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:09.038567 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f0cc8e-a551-4f4b-bbc8-dc5c75cc62b7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:09.046404 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f0cc8e-a551-4f4b-bbc8-dc5c75cc62b7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:44:09.048006 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f0cc8e-a551-4f4b-bbc8-dc5c75cc62b7 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:44:09.057177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 759/1527] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:44:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:44:13.090324 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17caf6b9-9bf5-47c4-babb-b00f853b6fbd None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:13.090922 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17caf6b9-9bf5-47c4-babb-b00f853b6fbd None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:13.090922 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17caf6b9-9bf5-47c4-babb-b00f853b6fbd None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:13.092238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 769/1528] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 30 09:44:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:44:13.097403 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3150253-dfdf-4b98-8d3e-d6bac2cb8dc7 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:13.097832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3150253-dfdf-4b98-8d3e-d6bac2cb8dc7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:13.097930 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3150253-dfdf-4b98-8d3e-d6bac2cb8dc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:13.117528 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b3150253-dfdf-4b98-8d3e-d6bac2cb8dc7 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:44:13.118754 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b3150253-dfdf-4b98-8d3e-d6bac2cb8dc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:44:13.126758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 760/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 30 09:44:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:44:13.414945 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17caf6b9-9bf5-47c4-babb-b00f853b6fbd None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:44:13.416636 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-17caf6b9-9bf5-47c4-babb-b00f853b6fbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:44:13.428403 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:44:13.473612 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17caf6b9-9bf5-47c4-babb-b00f853b6fbd None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:44:13.477168 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672c98f4-c869-4251-9b37-5580322c9df2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:44:13.477766 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672c98f4-c869-4251-9b37-5580322c9df2 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:13.478045 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672c98f4-c869-4251-9b37-5580322c9df2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:13.478318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672c98f4-c869-4251-9b37-5580322c9df2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:13.487971 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672c98f4-c869-4251-9b37-5580322c9df2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:44:13.489412 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672c98f4-c869-4251-9b37-5580322c9df2 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:44:13.498664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 770/1530] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:44:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:44:15.650868 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463c042c-3f65-4715-bef3-7712adb6d354 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:15.651431 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463c042c-3f65-4715-bef3-7712adb6d354 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:15.651431 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463c042c-3f65-4715-bef3-7712adb6d354 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:15.652755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 761/1531] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 30 09:44:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:44:15.657817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c904bc59-0807-4d62-8b5d-fcd94f8a6b9d None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:15.658788 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c904bc59-0807-4d62-8b5d-fcd94f8a6b9d None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:15.658885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c904bc59-0807-4d62-8b5d-fcd94f8a6b9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:15.678492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-c904bc59-0807-4d62-8b5d-fcd94f8a6b9d None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:44:15.679705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c904bc59-0807-4d62-8b5d-fcd94f8a6b9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:44:15.687476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 771/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 30 09:44:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:44:15.973746 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463c042c-3f65-4715-bef3-7712adb6d354 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:44:15.984387 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463c042c-3f65-4715-bef3-7712adb6d354 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:44:15.988478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8c7b3a-124e-48ed-8f47-2f52ec7478b7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:44:15.992331 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8c7b3a-124e-48ed-8f47-2f52ec7478b7 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:15.992331 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8c7b3a-124e-48ed-8f47-2f52ec7478b7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:15.992331 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8c7b3a-124e-48ed-8f47-2f52ec7478b7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:15.998198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8c7b3a-124e-48ed-8f47-2f52ec7478b7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:44:15.999954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8c7b3a-124e-48ed-8f47-2f52ec7478b7 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:44:16.009147 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 762/1533] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:44:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:44:19.698394 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada69d18-b489-4fba-be80-a458601db0dc None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:19.699030 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada69d18-b489-4fba-be80-a458601db0dc None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:19.699030 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada69d18-b489-4fba-be80-a458601db0dc None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:19.700300 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 772/1534] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 30 09:44:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:44:19.705991 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c63fd4-0930-4359-986e-d0122dce06cc None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:19.710085 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c63fd4-0930-4359-986e-d0122dce06cc None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:19.710085 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c63fd4-0930-4359-986e-d0122dce06cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:19.714480 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=74 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:44:19.726377 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c4c63fd4-0930-4359-986e-d0122dce06cc None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:44:19.727641 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c4c63fd4-0930-4359-986e-d0122dce06cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:44:19.735586 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 763/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 30 09:44:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:44:20.036231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada69d18-b489-4fba-be80-a458601db0dc None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:44:20.037914 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ada69d18-b489-4fba-be80-a458601db0dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:44:20.094242 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada69d18-b489-4fba-be80-a458601db0dc None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:44:20.097903 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6818a4af-189e-4ba2-af1e-291ce15bb81f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:44:20.098495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6818a4af-189e-4ba2-af1e-291ce15bb81f service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:20.098800 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6818a4af-189e-4ba2-af1e-291ce15bb81f service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:20.099155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6818a4af-189e-4ba2-af1e-291ce15bb81f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:20.107832 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6818a4af-189e-4ba2-af1e-291ce15bb81f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:44:20.109301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6818a4af-189e-4ba2-af1e-291ce15bb81f service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:44:20.119236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 773/1536] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:44:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:44:22.267090 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f1a9ff-5d41-4c22-b403-a29bb600c350 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:22.267621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f1a9ff-5d41-4c22-b403-a29bb600c350 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:22.267621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f1a9ff-5d41-4c22-b403-a29bb600c350 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:22.268977 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 764/1537] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 30 09:44:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:44:22.274121 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2777cb8e-10f0-47c9-a528-57513f9d7182 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:22.274572 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2777cb8e-10f0-47c9-a528-57513f9d7182 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:22.274695 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2777cb8e-10f0-47c9-a528-57513f9d7182 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:22.293875 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-2777cb8e-10f0-47c9-a528-57513f9d7182 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:44:22.295092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-2777cb8e-10f0-47c9-a528-57513f9d7182 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:44:22.303276 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 774/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 30 09:44:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:44:22.601956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f1a9ff-5d41-4c22-b403-a29bb600c350 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:44:22.603668 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-81f1a9ff-5d41-4c22-b403-a29bb600c350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:44:22.616980 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f1a9ff-5d41-4c22-b403-a29bb600c350 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:44:22.620427 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76c2ea9-b840-406c-9703-c6160304bde6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:44:22.621015 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76c2ea9-b840-406c-9703-c6160304bde6 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:44:22.621291 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76c2ea9-b840-406c-9703-c6160304bde6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:44:22.621559 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76c2ea9-b840-406c-9703-c6160304bde6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:44:22.630573 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76c2ea9-b840-406c-9703-c6160304bde6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:44:22.632019 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76c2ea9-b840-406c-9703-c6160304bde6 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:44:22.641150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 765/1539] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:44:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:44:32.289217 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=54 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:44:32.637091 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=8 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:44.377801 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7573d9-154a-4952-bca0-884718a26b5c None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:44.379003 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7573d9-154a-4952-bca0-884718a26b5c None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:44.379531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7573d9-154a-4952-bca0-884718a26b5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:44.409019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-ce7573d9-154a-4952-bca0-884718a26b5c None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:44.410241 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ce7573d9-154a-4952-bca0-884718a26b5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:44.422331 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 775/1540] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:44.433302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12fdf3f-ea9f-49a1-b6a5-83947394897b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:44.433921 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12fdf3f-ea9f-49a1-b6a5-83947394897b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:44.433921 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12fdf3f-ea9f-49a1-b6a5-83947394897b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:44.460547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-a12fdf3f-ea9f-49a1-b6a5-83947394897b None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:44.461812 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a12fdf3f-ea9f-49a1-b6a5-83947394897b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:44.475502 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 766/1541] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:44.486141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7573d9-154a-4952-bca0-884718a26b5c None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:44.501621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7573d9-154a-4952-bca0-884718a26b5c None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:44.517168 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e294433-146a-4651-8f9f-9551b34f9536 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:44.518182 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e294433-146a-4651-8f9f-9551b34f9536 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:44.518864 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e294433-146a-4651-8f9f-9551b34f9536 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:44.519360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e294433-146a-4651-8f9f-9551b34f9536 admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:44.520272 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e294433-146a-4651-8f9f-9551b34f9536 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:45:44.520782 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e294433-146a-4651-8f9f-9551b34f9536 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:44.524025 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e294433-146a-4651-8f9f-9551b34f9536 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:44.540429 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 776/1542] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:45:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:45:44.549789 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12fdf3f-ea9f-49a1-b6a5-83947394897b None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:44.561730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12fdf3f-ea9f-49a1-b6a5-83947394897b None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:44.571495 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f140ac3a-2839-418b-92fe-0126b6b6d8cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:44.572144 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f140ac3a-2839-418b-92fe-0126b6b6d8cc admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:44.572427 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f140ac3a-2839-418b-92fe-0126b6b6d8cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:44.572702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f140ac3a-2839-418b-92fe-0126b6b6d8cc admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:44.573363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f140ac3a-2839-418b-92fe-0126b6b6d8cc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:44.575379 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f140ac3a-2839-418b-92fe-0126b6b6d8cc admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:44.687981 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b445de-5c9c-4a4b-96ef-cb9a3984dc2f None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:44.688827 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b445de-5c9c-4a4b-96ef-cb9a3984dc2f None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:44.689129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b445de-5c9c-4a4b-96ef-cb9a3984dc2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:44.715965 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 767/1543] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:45:44 2022] POST /identity/v3/projects => generated 391 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:44.720390 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f140ac3a-2839-418b-92fe-0126b6b6d8cc admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:44.729951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-e5b445de-5c9c-4a4b-96ef-cb9a3984dc2f None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:44.733338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-e5b445de-5c9c-4a4b-96ef-cb9a3984dc2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:44.848433 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f140ac3a-2839-418b-92fe-0126b6b6d8cc admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:44.858662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 777/1544] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:45:44.863574 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb54d42-2f53-48de-b82c-2590364cc042 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:44.863717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42094d28-0df7-4386-a603-f382393772cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:44.863846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb54d42-2f53-48de-b82c-2590364cc042 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:44.863846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb54d42-2f53-48de-b82c-2590364cc042 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:44.864045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42094d28-0df7-4386-a603-f382393772cc admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:44.864045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42094d28-0df7-4386-a603-f382393772cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:44.864045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42094d28-0df7-4386-a603-f382393772cc admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:44.864045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42094d28-0df7-4386-a603-f382393772cc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:44.864045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: WARNING py.warnings [None req-42094d28-0df7-4386-a603-f382393772cc admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 30 09:45:44.864045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: warnings.warn(msg) Sep 30 09:45:44.864045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: Sep 30 09:45:44.864655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42094d28-0df7-4386-a603-f382393772cc admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:44.902409 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-feb54d42-2f53-48de-b82c-2590364cc042 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:44.903731 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-feb54d42-2f53-48de-b82c-2590364cc042 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:44.912364 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 778/1545] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:44.913234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3babe5-56bf-439f-b2ff-95e4ae24a4bc None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:44.913517 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3babe5-56bf-439f-b2ff-95e4ae24a4bc None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:44.913786 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3babe5-56bf-439f-b2ff-95e4ae24a4bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:44.933995 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-5d3babe5-56bf-439f-b2ff-95e4ae24a4bc None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:44.935241 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-5d3babe5-56bf-439f-b2ff-95e4ae24a4bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:44.944085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 779/1546] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:44.945463 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbe5e88-7bf8-4d41-9c43-ee6491ad9bf6 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:44.945750 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbe5e88-7bf8-4d41-9c43-ee6491ad9bf6 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:44.946021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbe5e88-7bf8-4d41-9c43-ee6491ad9bf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:44.965077 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 768/1547] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:45:44 2022] POST /identity/v3/users => generated 296 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:44.969487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-2fbe5e88-7bf8-4d41-9c43-ee6491ad9bf6 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:44.969646 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a978473-5eb0-4b08-b384-4aaee69fe5da None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:44.969646 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a978473-5eb0-4b08-b384-4aaee69fe5da None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:44.969646 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a978473-5eb0-4b08-b384-4aaee69fe5da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:44.969950 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-2fbe5e88-7bf8-4d41-9c43-ee6491ad9bf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:44.976107 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 780/1548] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:44.977462 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff833bfe-c55f-453b-b9b7-782a122fb28c None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:44.977744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff833bfe-c55f-453b-b9b7-782a122fb28c None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:44.978011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff833bfe-c55f-453b-b9b7-782a122fb28c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:44.991536 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9a978473-5eb0-4b08-b384-4aaee69fe5da None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:44.992751 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9a978473-5eb0-4b08-b384-4aaee69fe5da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:44.997548 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-ff833bfe-c55f-453b-b9b7-782a122fb28c None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:44.998746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ff833bfe-c55f-453b-b9b7-782a122fb28c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:45.001341 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 769/1549] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:45.002498 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072386cf-04e7-44b1-a242-9ee408952161 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.004113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072386cf-04e7-44b1-a242-9ee408952161 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.004113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072386cf-04e7-44b1-a242-9ee408952161 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.014645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 781/1550] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:45.014645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0c71dc-3ea5-4ad8-9bd5-c624a855005b None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.014645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0c71dc-3ea5-4ad8-9bd5-c624a855005b None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.014645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0c71dc-3ea5-4ad8-9bd5-c624a855005b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.022831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-072386cf-04e7-44b1-a242-9ee408952161 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:45.024021 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-072386cf-04e7-44b1-a242-9ee408952161 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:45.030019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-2a0c71dc-3ea5-4ad8-9bd5-c624a855005b None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:45.031228 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-2a0c71dc-3ea5-4ad8-9bd5-c624a855005b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:45.031731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 770/1551] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:45.032540 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072386cf-04e7-44b1-a242-9ee408952161 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:45.039203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 782/1552] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:45.040380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af478cf4-2825-41e3-80a4-671b7f07eb23 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.040669 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af478cf4-2825-41e3-80a4-671b7f07eb23 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.040947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af478cf4-2825-41e3-80a4-671b7f07eb23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.048902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072386cf-04e7-44b1-a242-9ee408952161 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:45.053501 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df814d3-d8d5-4505-b19d-b7da5e8c1f9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:45.054095 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df814d3-d8d5-4505-b19d-b7da5e8c1f9d admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.054397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df814d3-d8d5-4505-b19d-b7da5e8c1f9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.054701 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df814d3-d8d5-4505-b19d-b7da5e8c1f9d admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.055335 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df814d3-d8d5-4505-b19d-b7da5e8c1f9d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:45:45.055636 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df814d3-d8d5-4505-b19d-b7da5e8c1f9d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:45.057078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: WARNING py.warnings [None req-7df814d3-d8d5-4505-b19d-b7da5e8c1f9d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 30 09:45:45.057078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: warnings.warn(msg) Sep 30 09:45:45.057078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: Sep 30 09:45:45.057960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df814d3-d8d5-4505-b19d-b7da5e8c1f9d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:45.060214 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-af478cf4-2825-41e3-80a4-671b7f07eb23 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:45.061417 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-af478cf4-2825-41e3-80a4-671b7f07eb23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:45.069926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 783/1553] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:45.076481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 771/1554] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:45:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:45:45.076654 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af478cf4-2825-41e3-80a4-671b7f07eb23 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:45.076793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1698ec2-394c-4bb6-bf81-e424d042aaff None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.076793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1698ec2-394c-4bb6-bf81-e424d042aaff None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.076793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1698ec2-394c-4bb6-bf81-e424d042aaff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.087783 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af478cf4-2825-41e3-80a4-671b7f07eb23 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:45.092716 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad2645b-9575-4d9e-a8db-5367653cff83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:45.093312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad2645b-9575-4d9e-a8db-5367653cff83 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.093596 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad2645b-9575-4d9e-a8db-5367653cff83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.093870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad2645b-9575-4d9e-a8db-5367653cff83 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.093968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b1698ec2-394c-4bb6-bf81-e424d042aaff None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:45.094539 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad2645b-9575-4d9e-a8db-5367653cff83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:45.095168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b1698ec2-394c-4bb6-bf81-e424d042aaff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:45.096191 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad2645b-9575-4d9e-a8db-5367653cff83 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:45.102887 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 772/1555] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:45.104034 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ed83d4-afce-4417-8a0f-74b9d813850b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.104314 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ed83d4-afce-4417-8a0f-74b9d813850b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.104585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ed83d4-afce-4417-8a0f-74b9d813850b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.107626 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 784/1556] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:45 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:45.109256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85097b4-9255-4a78-bfa8-eccf7c0b4f9c None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.109542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85097b4-9255-4a78-bfa8-eccf7c0b4f9c None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.109813 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85097b4-9255-4a78-bfa8-eccf7c0b4f9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.123852 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-15ed83d4-afce-4417-8a0f-74b9d813850b None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:45.125053 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-15ed83d4-afce-4417-8a0f-74b9d813850b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:45.130405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-c85097b4-9255-4a78-bfa8-eccf7c0b4f9c None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:45.131595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c85097b4-9255-4a78-bfa8-eccf7c0b4f9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:45.133675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 773/1557] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:45.134446 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ed83d4-afce-4417-8a0f-74b9d813850b None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:45.139418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 785/1558] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:45.140933 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84f6eaf-0a73-40df-9794-b14c8cc56d7a None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.140933 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84f6eaf-0a73-40df-9794-b14c8cc56d7a None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.141134 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84f6eaf-0a73-40df-9794-b14c8cc56d7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.145232 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ed83d4-afce-4417-8a0f-74b9d813850b None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:45.149860 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60cc19f-a24d-4cb5-933c-d85559cff24d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:45.150458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60cc19f-a24d-4cb5-933c-d85559cff24d admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.150777 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60cc19f-a24d-4cb5-933c-d85559cff24d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.151056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60cc19f-a24d-4cb5-933c-d85559cff24d admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.151691 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60cc19f-a24d-4cb5-933c-d85559cff24d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:45:45.151995 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60cc19f-a24d-4cb5-933c-d85559cff24d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:45.153605 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60cc19f-a24d-4cb5-933c-d85559cff24d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:45.160505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-a84f6eaf-0a73-40df-9794-b14c8cc56d7a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:45.161688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a84f6eaf-0a73-40df-9794-b14c8cc56d7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:45.165441 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 774/1559] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:45:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:45:45.166618 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2775f559-71d3-4a57-a73a-be9abc32e84d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.166939 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2775f559-71d3-4a57-a73a-be9abc32e84d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.167216 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2775f559-71d3-4a57-a73a-be9abc32e84d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.171227 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 786/1560] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:45.175668 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84f6eaf-0a73-40df-9794-b14c8cc56d7a None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:45.186672 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84f6eaf-0a73-40df-9794-b14c8cc56d7a None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:45.186953 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-2775f559-71d3-4a57-a73a-be9abc32e84d None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:45.188681 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2775f559-71d3-4a57-a73a-be9abc32e84d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:45.191874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae486250-160b-4ba5-852c-ae9e20bac92e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:45.192526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae486250-160b-4ba5-852c-ae9e20bac92e admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.192816 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae486250-160b-4ba5-852c-ae9e20bac92e admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.193152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae486250-160b-4ba5-852c-ae9e20bac92e admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.193818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae486250-160b-4ba5-852c-ae9e20bac92e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:45.194870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: WARNING py.warnings [None req-ae486250-160b-4ba5-852c-ae9e20bac92e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 30 09:45:45.194870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: warnings.warn(msg) Sep 30 09:45:45.194870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: Sep 30 09:45:45.196116 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae486250-160b-4ba5-852c-ae9e20bac92e admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:45.196808 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 775/1561] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:45.197404 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2775f559-71d3-4a57-a73a-be9abc32e84d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:45.207421 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2775f559-71d3-4a57-a73a-be9abc32e84d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:45.212021 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40ecb29-4340-4200-b42d-7b875dee46bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:45.212614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40ecb29-4340-4200-b42d-7b875dee46bc admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.212904 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40ecb29-4340-4200-b42d-7b875dee46bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.213182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40ecb29-4340-4200-b42d-7b875dee46bc admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.213817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40ecb29-4340-4200-b42d-7b875dee46bc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:45:45.214123 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40ecb29-4340-4200-b42d-7b875dee46bc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:45.215768 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40ecb29-4340-4200-b42d-7b875dee46bc admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:45.227420 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 776/1562] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:45:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:45:45.228199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40ecb29-4340-4200-b42d-7b875dee46bc admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:45.238340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40ecb29-4340-4200-b42d-7b875dee46bc admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:45.243272 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2c58ba-8e41-486a-8dc5-a42213c5f191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:45.243877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2c58ba-8e41-486a-8dc5-a42213c5f191 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.244160 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2c58ba-8e41-486a-8dc5-a42213c5f191 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.244436 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2c58ba-8e41-486a-8dc5-a42213c5f191 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.245066 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2c58ba-8e41-486a-8dc5-a42213c5f191 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:45.248174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2c58ba-8e41-486a-8dc5-a42213c5f191 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:45.259487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 777/1563] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:45 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:45.260992 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2c58ba-8e41-486a-8dc5-a42213c5f191 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:45.270708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2c58ba-8e41-486a-8dc5-a42213c5f191 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:45.275213 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d697c6-b5da-4c56-b3f1-71dfb61f4346 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:45.275778 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d697c6-b5da-4c56-b3f1-71dfb61f4346 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.278689 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d697c6-b5da-4c56-b3f1-71dfb61f4346 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.278689 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d697c6-b5da-4c56-b3f1-71dfb61f4346 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.278689 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d697c6-b5da-4c56-b3f1-71dfb61f4346 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:45:45.278689 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d697c6-b5da-4c56-b3f1-71dfb61f4346 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:45.279039 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d697c6-b5da-4c56-b3f1-71dfb61f4346 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:45.289866 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 778/1564] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:45:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:45:45.290632 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d697c6-b5da-4c56-b3f1-71dfb61f4346 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:45.302520 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d697c6-b5da-4c56-b3f1-71dfb61f4346 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:45.304368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 787/1565] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:45:45 2022] POST /identity/v3/projects => generated 363 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:45.305235 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae486250-160b-4ba5-852c-ae9e20bac92e admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:45.315556 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:45.359674 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b95ee6d-1e4a-41f1-a5e3-a9ce3cdf4b25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:45.360786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b95ee6d-1e4a-41f1-a5e3-a9ce3cdf4b25 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.361064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b95ee6d-1e4a-41f1-a5e3-a9ce3cdf4b25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.361336 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b95ee6d-1e4a-41f1-a5e3-a9ce3cdf4b25 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.362005 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b95ee6d-1e4a-41f1-a5e3-a9ce3cdf4b25 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:45:45.362312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b95ee6d-1e4a-41f1-a5e3-a9ce3cdf4b25 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:45.363393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:45.363960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b95ee6d-1e4a-41f1-a5e3-a9ce3cdf4b25 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:45.368318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae486250-160b-4ba5-852c-ae9e20bac92e admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:45.373354 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11528df1-b4dd-406b-a961-b1ecb8b1b35f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:45.376424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11528df1-b4dd-406b-a961-b1ecb8b1b35f admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.376424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11528df1-b4dd-406b-a961-b1ecb8b1b35f admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.376424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11528df1-b4dd-406b-a961-b1ecb8b1b35f admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.376424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11528df1-b4dd-406b-a961-b1ecb8b1b35f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:45.377011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 779/1566] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:45:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:45:45.377793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b95ee6d-1e4a-41f1-a5e3-a9ce3cdf4b25 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:45.379578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11528df1-b4dd-406b-a961-b1ecb8b1b35f admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:45.387542 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b95ee6d-1e4a-41f1-a5e3-a9ce3cdf4b25 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:45.392274 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e02347f-d272-411c-88eb-e1fbf006ce46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:45.392892 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e02347f-d272-411c-88eb-e1fbf006ce46 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.393184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e02347f-d272-411c-88eb-e1fbf006ce46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.393440 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e02347f-d272-411c-88eb-e1fbf006ce46 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.394070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e02347f-d272-411c-88eb-e1fbf006ce46 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:45:45.394381 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e02347f-d272-411c-88eb-e1fbf006ce46 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:45.395990 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e02347f-d272-411c-88eb-e1fbf006ce46 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:45.407888 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 780/1567] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:45:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:45:45.408655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e02347f-d272-411c-88eb-e1fbf006ce46 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:45.419359 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e02347f-d272-411c-88eb-e1fbf006ce46 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:45.424560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9848a6-08c6-4be2-8dc3-fd84e7c5cebe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:45.425125 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9848a6-08c6-4be2-8dc3-fd84e7c5cebe admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.425403 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9848a6-08c6-4be2-8dc3-fd84e7c5cebe admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.425674 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9848a6-08c6-4be2-8dc3-fd84e7c5cebe admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.426286 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9848a6-08c6-4be2-8dc3-fd84e7c5cebe admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:45:45.426647 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9848a6-08c6-4be2-8dc3-fd84e7c5cebe admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:45.432177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9848a6-08c6-4be2-8dc3-fd84e7c5cebe admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:45.439755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 781/1568] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:45:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:45:45.440694 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9848a6-08c6-4be2-8dc3-fd84e7c5cebe admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:45.449457 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9848a6-08c6-4be2-8dc3-fd84e7c5cebe admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:45.454005 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210890b6-7a09-4bf7-b4f3-fc97d6735712 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:45.455166 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210890b6-7a09-4bf7-b4f3-fc97d6735712 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.455166 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210890b6-7a09-4bf7-b4f3-fc97d6735712 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.455166 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210890b6-7a09-4bf7-b4f3-fc97d6735712 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.455701 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210890b6-7a09-4bf7-b4f3-fc97d6735712 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:45.457531 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210890b6-7a09-4bf7-b4f3-fc97d6735712 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:45.483852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 788/1569] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:45:45 2022] POST /identity/v3/projects => generated 369 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:45.483852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11528df1-b4dd-406b-a961-b1ecb8b1b35f admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:45.540457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11528df1-b4dd-406b-a961-b1ecb8b1b35f admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:45.545710 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4050964a-5611-4f5c-9104-4212ebc4335a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:45.546366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4050964a-5611-4f5c-9104-4212ebc4335a admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.546665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4050964a-5611-4f5c-9104-4212ebc4335a admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.546937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4050964a-5611-4f5c-9104-4212ebc4335a admin admin] PATH_INFO: `/v3/projects/f1c0948ddb014e6f93d6069bb5203890/users/7ad21da05fc246d489b0afa641f20216/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.561200 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4050964a-5611-4f5c-9104-4212ebc4335a admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1c0948ddb014e6f93d6069bb5203890, user_id=7ad21da05fc246d489b0afa641f20216, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:45.562271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: WARNING py.warnings [None req-4050964a-5611-4f5c-9104-4212ebc4335a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 30 09:45:45.562271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: warnings.warn(msg) Sep 30 09:45:45.562271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: Sep 30 09:45:45.564890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4050964a-5611-4f5c-9104-4212ebc4335a admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:45.609593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 782/1570] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:45:45 2022] POST /identity/v3/projects => generated 373 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:45.609593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210890b6-7a09-4bf7-b4f3-fc97d6735712 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:45.633900 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 789/1571] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:45:45 2022] PUT /identity/v3/projects/f1c0948ddb014e6f93d6069bb5203890/users/7ad21da05fc246d489b0afa641f20216/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:45:45.635186 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4050964a-5611-4f5c-9104-4212ebc4335a admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:45.669584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210890b6-7a09-4bf7-b4f3-fc97d6735712 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:45.704504 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4050964a-5611-4f5c-9104-4212ebc4335a admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:45.709940 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30444dfb-b3ec-4478-b4eb-e428d0a3a9e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:45.710708 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30444dfb-b3ec-4478-b4eb-e428d0a3a9e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.711106 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30444dfb-b3ec-4478-b4eb-e428d0a3a9e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.711470 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30444dfb-b3ec-4478-b4eb-e428d0a3a9e7 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.712194 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30444dfb-b3ec-4478-b4eb-e428d0a3a9e7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:45.714502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30444dfb-b3ec-4478-b4eb-e428d0a3a9e7 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:45.726968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b84a75-2476-47b5-8172-05434cdcb7c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:45.727582 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b84a75-2476-47b5-8172-05434cdcb7c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.727859 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b84a75-2476-47b5-8172-05434cdcb7c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.728133 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b84a75-2476-47b5-8172-05434cdcb7c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.728760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b84a75-2476-47b5-8172-05434cdcb7c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:45.730683 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b84a75-2476-47b5-8172-05434cdcb7c6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:45.745519 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:45.827935 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 790/1572] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:45:45 2022] POST /identity/v3/users => generated 282 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:45.829559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30444dfb-b3ec-4478-b4eb-e428d0a3a9e7 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:45.839093 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30444dfb-b3ec-4478-b4eb-e428d0a3a9e7 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:45.843768 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbbc6358-90db-4eab-b3cf-bbfd849946a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:45.844332 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbc6358-90db-4eab-b3cf-bbfd849946a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.844608 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbc6358-90db-4eab-b3cf-bbfd849946a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.844879 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbc6358-90db-4eab-b3cf-bbfd849946a1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.845480 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbbc6358-90db-4eab-b3cf-bbfd849946a1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:45.847488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbbc6358-90db-4eab-b3cf-bbfd849946a1 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:45.863608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 783/1573] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:45:45 2022] POST /identity/v3/projects => generated 367 bytes in 256 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:45.863608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b84a75-2476-47b5-8172-05434cdcb7c6 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:45.924092 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b84a75-2476-47b5-8172-05434cdcb7c6 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:45.928700 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7465dc05-e759-4544-8656-e23e86e42e30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:45.929301 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7465dc05-e759-4544-8656-e23e86e42e30 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:45.929580 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7465dc05-e759-4544-8656-e23e86e42e30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:45.929856 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7465dc05-e759-4544-8656-e23e86e42e30 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:45.930482 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7465dc05-e759-4544-8656-e23e86e42e30 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:45.932413 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7465dc05-e759-4544-8656-e23e86e42e30 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:45.955224 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 791/1574] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:45:45 2022] POST /identity/v3/projects => generated 385 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:45.956167 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbbc6358-90db-4eab-b3cf-bbfd849946a1 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:45.988721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:46.016761 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbbc6358-90db-4eab-b3cf-bbfd849946a1 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.021307 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346f8740-f3dc-4389-b4c1-92ce3d60d28c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.021892 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346f8740-f3dc-4389-b4c1-92ce3d60d28c admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.022170 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346f8740-f3dc-4389-b4c1-92ce3d60d28c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.022441 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346f8740-f3dc-4389-b4c1-92ce3d60d28c admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.023082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346f8740-f3dc-4389-b4c1-92ce3d60d28c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.024981 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346f8740-f3dc-4389-b4c1-92ce3d60d28c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.034622 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 784/1575] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:45:45 2022] POST /identity/v3/projects => generated 379 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:46.035407 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7465dc05-e759-4544-8656-e23e86e42e30 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.097485 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7465dc05-e759-4544-8656-e23e86e42e30 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.102203 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914b9be4-d7f4-4bf6-96f2-9f9cc7cab5a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.102830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914b9be4-d7f4-4bf6-96f2-9f9cc7cab5a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.103117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914b9be4-d7f4-4bf6-96f2-9f9cc7cab5a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.103396 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914b9be4-d7f4-4bf6-96f2-9f9cc7cab5a6 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.104023 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914b9be4-d7f4-4bf6-96f2-9f9cc7cab5a6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.106029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914b9be4-d7f4-4bf6-96f2-9f9cc7cab5a6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.121097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 792/1576] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:45:45 2022] POST /identity/v3/projects => generated 379 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:46.127512 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346f8740-f3dc-4389-b4c1-92ce3d60d28c admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.182617 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346f8740-f3dc-4389-b4c1-92ce3d60d28c admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.187332 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029efda8-59a4-46f5-ba9b-e437782ecf0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.190664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 785/1577] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:45:46 2022] POST /identity/v3/users => generated 285 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:46.190814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029efda8-59a4-46f5-ba9b-e437782ecf0c admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.190814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029efda8-59a4-46f5-ba9b-e437782ecf0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.190814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029efda8-59a4-46f5-ba9b-e437782ecf0c admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.191088 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b4d302-8503-46b2-be19-8458430e8c68 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.191088 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b4d302-8503-46b2-be19-8458430e8c68 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.191266 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029efda8-59a4-46f5-ba9b-e437782ecf0c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.191358 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b4d302-8503-46b2-be19-8458430e8c68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.191447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029efda8-59a4-46f5-ba9b-e437782ecf0c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.209273 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-29b4d302-8503-46b2-be19-8458430e8c68 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:46.213644 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-29b4d302-8503-46b2-be19-8458430e8c68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:46.218075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 786/1578] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:46.218865 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b4d302-8503-46b2-be19-8458430e8c68 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.229357 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b4d302-8503-46b2-be19-8458430e8c68 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.233898 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd578a7d-2132-4d90-8c2b-f327e67511f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.234525 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd578a7d-2132-4d90-8c2b-f327e67511f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.242331 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd578a7d-2132-4d90-8c2b-f327e67511f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.242331 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd578a7d-2132-4d90-8c2b-f327e67511f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.242331 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd578a7d-2132-4d90-8c2b-f327e67511f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.242331 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd578a7d-2132-4d90-8c2b-f327e67511f3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.247611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 787/1579] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:46 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:46.248478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd578a7d-2132-4d90-8c2b-f327e67511f3 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.257709 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd578a7d-2132-4d90-8c2b-f327e67511f3 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.262186 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074c6da2-d755-44a4-8901-4c9d1be79340 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.262755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074c6da2-d755-44a4-8901-4c9d1be79340 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.263032 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074c6da2-d755-44a4-8901-4c9d1be79340 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.263297 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074c6da2-d755-44a4-8901-4c9d1be79340 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.263886 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074c6da2-d755-44a4-8901-4c9d1be79340 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.266675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074c6da2-d755-44a4-8901-4c9d1be79340 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.307858 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 793/1580] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:45:46 2022] POST /identity/v3/users => generated 287 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:46.308609 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029efda8-59a4-46f5-ba9b-e437782ecf0c admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.318382 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029efda8-59a4-46f5-ba9b-e437782ecf0c admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.323007 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275b5858-0652-4eda-b1b7-4b016ab21f62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.323577 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275b5858-0652-4eda-b1b7-4b016ab21f62 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.323860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275b5858-0652-4eda-b1b7-4b016ab21f62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.324136 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275b5858-0652-4eda-b1b7-4b016ab21f62 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.324743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275b5858-0652-4eda-b1b7-4b016ab21f62 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.326683 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275b5858-0652-4eda-b1b7-4b016ab21f62 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.404851 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 788/1581] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:45:46 2022] POST /identity/v3/users => generated 284 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:46.405647 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074c6da2-d755-44a4-8901-4c9d1be79340 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.414903 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074c6da2-d755-44a4-8901-4c9d1be79340 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.419629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be0f584-4158-4db0-bc29-500bcd924395 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.420197 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be0f584-4158-4db0-bc29-500bcd924395 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.420479 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be0f584-4158-4db0-bc29-500bcd924395 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.420754 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be0f584-4158-4db0-bc29-500bcd924395 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.421358 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be0f584-4158-4db0-bc29-500bcd924395 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.423306 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be0f584-4158-4db0-bc29-500bcd924395 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.433500 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 794/1582] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:45:46 2022] POST /identity/v3/users => generated 293 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:46.434880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275b5858-0652-4eda-b1b7-4b016ab21f62 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.444922 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275b5858-0652-4eda-b1b7-4b016ab21f62 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.449503 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4583fb0c-b34f-4079-91f0-0de3d4640a94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.450089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4583fb0c-b34f-4079-91f0-0de3d4640a94 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.450371 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4583fb0c-b34f-4079-91f0-0de3d4640a94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.450676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4583fb0c-b34f-4079-91f0-0de3d4640a94 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.451292 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4583fb0c-b34f-4079-91f0-0de3d4640a94 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.453203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4583fb0c-b34f-4079-91f0-0de3d4640a94 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.525090 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 789/1583] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:45:46 2022] POST /identity/v3/users => generated 290 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:46.526049 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be0f584-4158-4db0-bc29-500bcd924395 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.535454 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be0f584-4158-4db0-bc29-500bcd924395 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.539939 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829407f5-cef8-4395-9c35-28b0c4aeeabe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.540583 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829407f5-cef8-4395-9c35-28b0c4aeeabe admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.540857 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829407f5-cef8-4395-9c35-28b0c4aeeabe admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.541128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829407f5-cef8-4395-9c35-28b0c4aeeabe admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.541746 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829407f5-cef8-4395-9c35-28b0c4aeeabe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.543365 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829407f5-cef8-4395-9c35-28b0c4aeeabe admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.553473 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 790/1584] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:46 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:46.554406 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829407f5-cef8-4395-9c35-28b0c4aeeabe admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.562374 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:INSERT=11,keystone:SELECT=89 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:46.562532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 795/1585] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:45:46 2022] POST /identity/v3/users => generated 290 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:46.563007 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4583fb0c-b34f-4079-91f0-0de3d4640a94 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.573319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4583fb0c-b34f-4079-91f0-0de3d4640a94 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.578119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6551504c-e70f-4cfa-afee-65ebd0799016 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.579499 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6551504c-e70f-4cfa-afee-65ebd0799016 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.579810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6551504c-e70f-4cfa-afee-65ebd0799016 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.580075 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6551504c-e70f-4cfa-afee-65ebd0799016 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.580744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6551504c-e70f-4cfa-afee-65ebd0799016 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.582311 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6551504c-e70f-4cfa-afee-65ebd0799016 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.592923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 796/1586] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:46.593914 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6551504c-e70f-4cfa-afee-65ebd0799016 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.604222 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6551504c-e70f-4cfa-afee-65ebd0799016 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.608932 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5c8518-428e-474e-a778-4d7f4ee15dc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.609560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5c8518-428e-474e-a778-4d7f4ee15dc8 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.609856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5c8518-428e-474e-a778-4d7f4ee15dc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.610140 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5c8518-428e-474e-a778-4d7f4ee15dc8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.610838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5c8518-428e-474e-a778-4d7f4ee15dc8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.612471 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5c8518-428e-474e-a778-4d7f4ee15dc8 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.612718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829407f5-cef8-4395-9c35-28b0c4aeeabe admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.616457 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae684d95-dde9-479f-8fdb-c700420ed553 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.617095 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae684d95-dde9-479f-8fdb-c700420ed553 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.617372 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae684d95-dde9-479f-8fdb-c700420ed553 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.617642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae684d95-dde9-479f-8fdb-c700420ed553 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.622976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 797/1587] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:46 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:46.623771 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5c8518-428e-474e-a778-4d7f4ee15dc8 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.630069 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae684d95-dde9-479f-8fdb-c700420ed553 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.632015 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae684d95-dde9-479f-8fdb-c700420ed553 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.633804 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5c8518-428e-474e-a778-4d7f4ee15dc8 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.638644 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cddac97-d566-4179-9970-626391645734 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.639369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cddac97-d566-4179-9970-626391645734 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.639770 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cddac97-d566-4179-9970-626391645734 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.640144 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cddac97-d566-4179-9970-626391645734 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.640956 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cddac97-d566-4179-9970-626391645734 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.642250 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 791/1588] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:45:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:45:46.643505 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae684d95-dde9-479f-8fdb-c700420ed553 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.643660 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cddac97-d566-4179-9970-626391645734 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.652745 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae684d95-dde9-479f-8fdb-c700420ed553 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.655725 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 798/1589] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:46 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:46.658724 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8868aa-1e7c-478a-8848-ea97409c0aba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.659588 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8868aa-1e7c-478a-8848-ea97409c0aba admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.659802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cddac97-d566-4179-9970-626391645734 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.659916 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8868aa-1e7c-478a-8848-ea97409c0aba admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.660104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8868aa-1e7c-478a-8848-ea97409c0aba admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.660919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8868aa-1e7c-478a-8848-ea97409c0aba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.662621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8868aa-1e7c-478a-8848-ea97409c0aba admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.669685 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cddac97-d566-4179-9970-626391645734 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.673065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 792/1590] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:46.677280 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8868aa-1e7c-478a-8848-ea97409c0aba admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.677405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a6c2e6-fd3f-49c5-920b-2e8e6534d2f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.677405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a6c2e6-fd3f-49c5-920b-2e8e6534d2f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.677405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a6c2e6-fd3f-49c5-920b-2e8e6534d2f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.677405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a6c2e6-fd3f-49c5-920b-2e8e6534d2f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.678078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a6c2e6-fd3f-49c5-920b-2e8e6534d2f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.679861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a6c2e6-fd3f-49c5-920b-2e8e6534d2f1 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.683506 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8868aa-1e7c-478a-8848-ea97409c0aba admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.688752 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f15264-e616-42b0-995c-300c6acde656 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.689339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f15264-e616-42b0-995c-300c6acde656 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.689619 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f15264-e616-42b0-995c-300c6acde656 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.689890 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f15264-e616-42b0-995c-300c6acde656 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.690535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f15264-e616-42b0-995c-300c6acde656 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.693995 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 799/1591] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:46 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:46.693995 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a6c2e6-fd3f-49c5-920b-2e8e6534d2f1 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.694202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f15264-e616-42b0-995c-300c6acde656 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.702079 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 793/1592] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:46 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:46.702877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f15264-e616-42b0-995c-300c6acde656 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.704843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a6c2e6-fd3f-49c5-920b-2e8e6534d2f1 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.710138 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c56553-d082-4420-a32f-7e0a41975933 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.710844 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c56553-d082-4420-a32f-7e0a41975933 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.716416 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c56553-d082-4420-a32f-7e0a41975933 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.716416 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c56553-d082-4420-a32f-7e0a41975933 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.716416 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c56553-d082-4420-a32f-7e0a41975933 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.716416 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c56553-d082-4420-a32f-7e0a41975933 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.720571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f15264-e616-42b0-995c-300c6acde656 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.725015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 800/1593] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:46 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:46.725245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7bc6a4-39c2-4027-b8f4-8b83f595bdba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.725657 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c56553-d082-4420-a32f-7e0a41975933 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.725890 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7bc6a4-39c2-4027-b8f4-8b83f595bdba admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.726172 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7bc6a4-39c2-4027-b8f4-8b83f595bdba admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.726446 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7bc6a4-39c2-4027-b8f4-8b83f595bdba admin admin] PATH_INFO: `/v3/projects/66fd4af7b1ba4f02bd6525daa760b783/users/76092190b13945b284c9bc6005bc5319/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.736328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c56553-d082-4420-a32f-7e0a41975933 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.741976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9902e0db-2b69-4b32-a96d-cedcb7cd1e4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.742140 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7bc6a4-39c2-4027-b8f4-8b83f595bdba admin admin] RBAC: Authorizing `identity:create_grant(project_id=66fd4af7b1ba4f02bd6525daa760b783, user_id=76092190b13945b284c9bc6005bc5319, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.742628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9902e0db-2b69-4b32-a96d-cedcb7cd1e4d admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.742902 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9902e0db-2b69-4b32-a96d-cedcb7cd1e4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.743181 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9902e0db-2b69-4b32-a96d-cedcb7cd1e4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.743829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9902e0db-2b69-4b32-a96d-cedcb7cd1e4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.744797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7bc6a4-39c2-4027-b8f4-8b83f595bdba admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.745430 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9902e0db-2b69-4b32-a96d-cedcb7cd1e4d admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.757537 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 801/1594] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:46 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:46.758367 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9902e0db-2b69-4b32-a96d-cedcb7cd1e4d admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.766756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=90,keystone:INSERT=10 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:46.770856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9902e0db-2b69-4b32-a96d-cedcb7cd1e4d admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.775562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713550d7-4052-433a-bcbf-0bf687720b47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.776152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713550d7-4052-433a-bcbf-0bf687720b47 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.776429 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713550d7-4052-433a-bcbf-0bf687720b47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.776707 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713550d7-4052-433a-bcbf-0bf687720b47 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.777333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713550d7-4052-433a-bcbf-0bf687720b47 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.778957 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713550d7-4052-433a-bcbf-0bf687720b47 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.788984 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 802/1595] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:46 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:46.790721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713550d7-4052-433a-bcbf-0bf687720b47 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.799654 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713550d7-4052-433a-bcbf-0bf687720b47 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.804468 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70af401f-1357-4636-90a1-0c152b8ee789 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.804793 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70af401f-1357-4636-90a1-0c152b8ee789 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.805071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70af401f-1357-4636-90a1-0c152b8ee789 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.805339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70af401f-1357-4636-90a1-0c152b8ee789 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.805957 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70af401f-1357-4636-90a1-0c152b8ee789 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.807595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70af401f-1357-4636-90a1-0c152b8ee789 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.818716 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 794/1596] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:45:46 2022] PUT /identity/v3/projects/66fd4af7b1ba4f02bd6525daa760b783/users/76092190b13945b284c9bc6005bc5319/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:45:46.819760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7bc6a4-39c2-4027-b8f4-8b83f595bdba admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.819988 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 803/1597] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:46.820881 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70af401f-1357-4636-90a1-0c152b8ee789 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.831258 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70af401f-1357-4636-90a1-0c152b8ee789 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.890379 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7bc6a4-39c2-4027-b8f4-8b83f595bdba admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.891403 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731ded60-a600-4e68-b6e7-91a0e8cf9a40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.892072 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731ded60-a600-4e68-b6e7-91a0e8cf9a40 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.892355 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731ded60-a600-4e68-b6e7-91a0e8cf9a40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.892630 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731ded60-a600-4e68-b6e7-91a0e8cf9a40 admin admin] PATH_INFO: `/v3/projects/15f01216eb594d88b37cb4377de3b831/users/f91b594c06c1445198c7285f6053a4f1/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.895411 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feefd638-f51a-4c75-822d-cfa08f8e2267 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.896079 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feefd638-f51a-4c75-822d-cfa08f8e2267 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.896299 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feefd638-f51a-4c75-822d-cfa08f8e2267 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.896583 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feefd638-f51a-4c75-822d-cfa08f8e2267 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.897252 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feefd638-f51a-4c75-822d-cfa08f8e2267 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.898943 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feefd638-f51a-4c75-822d-cfa08f8e2267 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.908214 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731ded60-a600-4e68-b6e7-91a0e8cf9a40 admin admin] RBAC: Authorizing `identity:create_grant(project_id=15f01216eb594d88b37cb4377de3b831, user_id=f91b594c06c1445198c7285f6053a4f1, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.912847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 795/1598] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:46 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:46.912847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feefd638-f51a-4c75-822d-cfa08f8e2267 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.913091 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731ded60-a600-4e68-b6e7-91a0e8cf9a40 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.923272 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feefd638-f51a-4c75-822d-cfa08f8e2267 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.928245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9c57c5-e81f-4d1b-89e8-215f77f81ad6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.928955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9c57c5-e81f-4d1b-89e8-215f77f81ad6 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.929329 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9c57c5-e81f-4d1b-89e8-215f77f81ad6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.929702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9c57c5-e81f-4d1b-89e8-215f77f81ad6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.930435 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9c57c5-e81f-4d1b-89e8-215f77f81ad6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.932353 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9c57c5-e81f-4d1b-89e8-215f77f81ad6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.943588 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 796/1599] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:46 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:46.947700 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9c57c5-e81f-4d1b-89e8-215f77f81ad6 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:46.958459 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9c57c5-e81f-4d1b-89e8-215f77f81ad6 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:46.974629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87efc4a1-c0a3-43fc-9b73-3ec549a3cde0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:46.974629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87efc4a1-c0a3-43fc-9b73-3ec549a3cde0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:46.974629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87efc4a1-c0a3-43fc-9b73-3ec549a3cde0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:46.974629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87efc4a1-c0a3-43fc-9b73-3ec549a3cde0 admin admin] PATH_INFO: `/v3/projects/d38d518c11a34ffc8921d4c241b8ce06/users/5a978f07591d4b78b248c27546a92b40/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:46.979971 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87efc4a1-c0a3-43fc-9b73-3ec549a3cde0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d38d518c11a34ffc8921d4c241b8ce06, user_id=5a978f07591d4b78b248c27546a92b40, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:46.982630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87efc4a1-c0a3-43fc-9b73-3ec549a3cde0 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:46.999719 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 804/1600] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:45:46 2022] PUT /identity/v3/projects/15f01216eb594d88b37cb4377de3b831/users/f91b594c06c1445198c7285f6053a4f1/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:45:47.002769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731ded60-a600-4e68-b6e7-91a0e8cf9a40 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:47.056245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 797/1601] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:45:46 2022] PUT /identity/v3/projects/d38d518c11a34ffc8921d4c241b8ce06/users/5a978f07591d4b78b248c27546a92b40/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:45:47.068513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87efc4a1-c0a3-43fc-9b73-3ec549a3cde0 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:47.071872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731ded60-a600-4e68-b6e7-91a0e8cf9a40 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:47.093378 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:47.130363 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bd4b7e-0acd-4298-8a20-9b811f7f4e65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:47.131103 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bd4b7e-0acd-4298-8a20-9b811f7f4e65 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:47.131465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bd4b7e-0acd-4298-8a20-9b811f7f4e65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:47.131747 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bd4b7e-0acd-4298-8a20-9b811f7f4e65 admin admin] PATH_INFO: `/v3/projects/a8f8c0108e1d45c6984325afff2c5fe9/users/1ce687dd153e4662a8bd34a0435d84e7/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:47.131884 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87efc4a1-c0a3-43fc-9b73-3ec549a3cde0 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:47.137265 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2def3f64-9aee-4799-a003-04f6c3a46425 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:47.137950 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2def3f64-9aee-4799-a003-04f6c3a46425 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:47.138225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2def3f64-9aee-4799-a003-04f6c3a46425 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:47.138494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2def3f64-9aee-4799-a003-04f6c3a46425 admin admin] PATH_INFO: `/v3/projects/86e6b0501cb94028be07bddd2182638b/users/8c4ced79c3ff4842b30b472acb555cd5/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:47.147413 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01bd4b7e-0acd-4298-8a20-9b811f7f4e65 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a8f8c0108e1d45c6984325afff2c5fe9, user_id=1ce687dd153e4662a8bd34a0435d84e7, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:47.151949 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01bd4b7e-0acd-4298-8a20-9b811f7f4e65 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:47.153771 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:47.154297 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2def3f64-9aee-4799-a003-04f6c3a46425 admin admin] RBAC: Authorizing `identity:create_grant(project_id=86e6b0501cb94028be07bddd2182638b, user_id=8c4ced79c3ff4842b30b472acb555cd5, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:47.157375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2def3f64-9aee-4799-a003-04f6c3a46425 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:47.263066 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 805/1602] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:45:46 2022] PUT /identity/v3/projects/a8f8c0108e1d45c6984325afff2c5fe9/users/1ce687dd153e4662a8bd34a0435d84e7/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:45:47.267468 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83294dce-10b6-4a68-81f8-1a6bad7200ab None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:47.267468 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83294dce-10b6-4a68-81f8-1a6bad7200ab None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:47.267468 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83294dce-10b6-4a68-81f8-1a6bad7200ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:47.277527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 798/1603] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:45:47 2022] PUT /identity/v3/projects/86e6b0501cb94028be07bddd2182638b/users/8c4ced79c3ff4842b30b472acb555cd5/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:45:47.282684 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2def3f64-9aee-4799-a003-04f6c3a46425 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:47.300763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-83294dce-10b6-4a68-81f8-1a6bad7200ab None None] MFA Rules not processed for user `76092190b13945b284c9bc6005bc5319`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:47.302122 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-83294dce-10b6-4a68-81f8-1a6bad7200ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:47.348551 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2def3f64-9aee-4799-a003-04f6c3a46425 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:47.353834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f87662f-f5c3-4434-af23-29d562f96c42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:47.354547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f87662f-f5c3-4434-af23-29d562f96c42 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:47.354864 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f87662f-f5c3-4434-af23-29d562f96c42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:47.355143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f87662f-f5c3-4434-af23-29d562f96c42 admin admin] PATH_INFO: `/v3/projects/b7f6b7ca2f3b4c20ab58940f7e24c1ed/users/ae7915e1b2474d9384a0fdcb2ea177b6/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:47.371938 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f87662f-f5c3-4434-af23-29d562f96c42 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b7f6b7ca2f3b4c20ab58940f7e24c1ed, user_id=ae7915e1b2474d9384a0fdcb2ea177b6, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:47.373127 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 806/1604] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:47 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:47.376626 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f87662f-f5c3-4434-af23-29d562f96c42 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:47.378045 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83294dce-10b6-4a68-81f8-1a6bad7200ab None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:47.389504 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83294dce-10b6-4a68-81f8-1a6bad7200ab None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:47.394376 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e203f532-d35c-4434-bf93-8e6a82aa98b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:47.395063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e203f532-d35c-4434-bf93-8e6a82aa98b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:47.395343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e203f532-d35c-4434-bf93-8e6a82aa98b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:47.395610 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e203f532-d35c-4434-bf93-8e6a82aa98b9 admin admin] PATH_INFO: `/v3/projects/c93c658159364808a249bdf8b2ccd752/users/ab91a478674f4bc2804029c299138ce5/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:47.411598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e203f532-d35c-4434-bf93-8e6a82aa98b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c93c658159364808a249bdf8b2ccd752, user_id=ab91a478674f4bc2804029c299138ce5, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:47.414260 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e203f532-d35c-4434-bf93-8e6a82aa98b9 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:47.433658 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 799/1605] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:45:47 2022] PUT /identity/v3/projects/b7f6b7ca2f3b4c20ab58940f7e24c1ed/users/ae7915e1b2474d9384a0fdcb2ea177b6/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:45:47.435022 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b426965f-2733-4913-905b-c6ab6eae02e2 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:47.439659 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b426965f-2733-4913-905b-c6ab6eae02e2 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:47.439659 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b426965f-2733-4913-905b-c6ab6eae02e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:47.463984 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b426965f-2733-4913-905b-c6ab6eae02e2 None None] MFA Rules not processed for user `f91b594c06c1445198c7285f6053a4f1`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:47.465211 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b426965f-2733-4913-905b-c6ab6eae02e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:47.498024 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 807/1606] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:45:47 2022] PUT /identity/v3/projects/c93c658159364808a249bdf8b2ccd752/users/ab91a478674f4bc2804029c299138ce5/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:45:47.499614 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebad419-16f7-42e6-9208-005216cc5e57 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:47.499906 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebad419-16f7-42e6-9208-005216cc5e57 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:47.500179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebad419-16f7-42e6-9208-005216cc5e57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:47.528313 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-6ebad419-16f7-42e6-9208-005216cc5e57 None None] MFA Rules not processed for user `5a978f07591d4b78b248c27546a92b40`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:47.529572 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-6ebad419-16f7-42e6-9208-005216cc5e57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:47.532529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 800/1607] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:47 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:47.533839 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181569d6-02ad-4395-8cf5-0e047b808939 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:47.534289 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181569d6-02ad-4395-8cf5-0e047b808939 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:47.534734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181569d6-02ad-4395-8cf5-0e047b808939 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:47.564539 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-181569d6-02ad-4395-8cf5-0e047b808939 None None] MFA Rules not processed for user `1ce687dd153e4662a8bd34a0435d84e7`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:47.565739 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-181569d6-02ad-4395-8cf5-0e047b808939 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:47.571529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:47.596009 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 808/1608] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:47 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:47.597258 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba6e12b-3961-496e-8e25-3c44cbac2dad None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:47.597543 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba6e12b-3961-496e-8e25-3c44cbac2dad None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:47.597840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba6e12b-3961-496e-8e25-3c44cbac2dad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:47.618373 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-4ba6e12b-3961-496e-8e25-3c44cbac2dad None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:47.619617 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-4ba6e12b-3961-496e-8e25-3c44cbac2dad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:47.643075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 801/1609] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:47 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:47.643075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181569d6-02ad-4395-8cf5-0e047b808939 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:47.677301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:47.692404 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 809/1610] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:47.694148 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e77a391-e164-4bd2-a55d-1ed417262f12 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:47.694431 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e77a391-e164-4bd2-a55d-1ed417262f12 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:47.694770 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e77a391-e164-4bd2-a55d-1ed417262f12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:47.714344 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-9e77a391-e164-4bd2-a55d-1ed417262f12 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:47.714813 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181569d6-02ad-4395-8cf5-0e047b808939 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:47.715756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-9e77a391-e164-4bd2-a55d-1ed417262f12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:47.720443 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364808a1-6de8-4518-a15a-7531809a60c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:47.721051 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364808a1-6de8-4518-a15a-7531809a60c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:47.721331 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364808a1-6de8-4518-a15a-7531809a60c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:47.721608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364808a1-6de8-4518-a15a-7531809a60c7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:47.722236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364808a1-6de8-4518-a15a-7531809a60c7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:47.723846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 810/1611] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:47.724186 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364808a1-6de8-4518-a15a-7531809a60c7 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:47.725003 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb0e12a-ee65-4eab-98d6-a28ff42be2db None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:47.725279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb0e12a-ee65-4eab-98d6-a28ff42be2db None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:47.725545 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb0e12a-ee65-4eab-98d6-a28ff42be2db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:47.753288 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-6fb0e12a-ee65-4eab-98d6-a28ff42be2db None None] MFA Rules not processed for user `ab91a478674f4bc2804029c299138ce5`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:47.754506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-6fb0e12a-ee65-4eab-98d6-a28ff42be2db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:47.817756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 811/1612] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:47 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:47.819487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb0e12a-ee65-4eab-98d6-a28ff42be2db None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:47.830796 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb0e12a-ee65-4eab-98d6-a28ff42be2db None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:47.835587 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1085097c-1f92-445c-a426-f65029f1934a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:47.836168 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1085097c-1f92-445c-a426-f65029f1934a admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:47.836448 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1085097c-1f92-445c-a426-f65029f1934a admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:47.836720 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1085097c-1f92-445c-a426-f65029f1934a admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:47.837335 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1085097c-1f92-445c-a426-f65029f1934a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:47.839294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1085097c-1f92-445c-a426-f65029f1934a admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:47.863824 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 802/1613] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:45:47 2022] POST /identity/v3/projects => generated 363 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:47.867332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364808a1-6de8-4518-a15a-7531809a60c7 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:47.920527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 812/1614] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:45:47 2022] POST /identity/v3/projects => generated 367 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:47.921941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1085097c-1f92-445c-a426-f65029f1934a admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:47.941112 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364808a1-6de8-4518-a15a-7531809a60c7 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:47.989590 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1085097c-1f92-445c-a426-f65029f1934a admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:47.992678 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:47.995527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56ae1a4-29bb-4476-b9a8-079e63241266 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:47.996641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56ae1a4-29bb-4476-b9a8-079e63241266 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:47.996937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56ae1a4-29bb-4476-b9a8-079e63241266 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:47.997471 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56ae1a4-29bb-4476-b9a8-079e63241266 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:48.012478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0acaf60-bf66-4cf8-9437-e6419338710c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:48.013098 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0acaf60-bf66-4cf8-9437-e6419338710c admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:48.022809 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0acaf60-bf66-4cf8-9437-e6419338710c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:48.022809 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0acaf60-bf66-4cf8-9437-e6419338710c admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:48.022809 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0acaf60-bf66-4cf8-9437-e6419338710c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:48.022809 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0acaf60-bf66-4cf8-9437-e6419338710c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:48.056627 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56ae1a4-29bb-4476-b9a8-079e63241266 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:48.058076 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56ae1a4-29bb-4476-b9a8-079e63241266 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:48.068031 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 813/1615] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:45:47 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:45:48.068860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56ae1a4-29bb-4476-b9a8-079e63241266 service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:48.078115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:48.111431 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 803/1616] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:45:47 2022] POST /identity/v3/projects => generated 373 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:48.115308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0acaf60-bf66-4cf8-9437-e6419338710c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:48.151130 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56ae1a4-29bb-4476-b9a8-079e63241266 service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:48.180995 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0acaf60-bf66-4cf8-9437-e6419338710c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:48.185239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0a5abe-a12d-458f-9ff9-d261a04213b5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:48.185987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0a5abe-a12d-458f-9ff9-d261a04213b5 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:48.186340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0a5abe-a12d-458f-9ff9-d261a04213b5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:48.186714 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0a5abe-a12d-458f-9ff9-d261a04213b5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:48.224998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a85d204-9ade-48b7-b06f-3e16f70a5d9b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:48.225669 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a85d204-9ade-48b7-b06f-3e16f70a5d9b service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:48.226047 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a85d204-9ade-48b7-b06f-3e16f70a5d9b service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:48.226506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a85d204-9ade-48b7-b06f-3e16f70a5d9b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:48.295426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0a5abe-a12d-458f-9ff9-d261a04213b5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:48.296997 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0a5abe-a12d-458f-9ff9-d261a04213b5 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:48.308100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 804/1617] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:45:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:45:48.308723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0a5abe-a12d-458f-9ff9-d261a04213b5 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:48.319093 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0a5abe-a12d-458f-9ff9-d261a04213b5 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:48.324075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf34c1a-954e-40ba-af90-af7f918b2ae3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:48.325308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf34c1a-954e-40ba-af90-af7f918b2ae3 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:48.325308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf34c1a-954e-40ba-af90-af7f918b2ae3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:48.325308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf34c1a-954e-40ba-af90-af7f918b2ae3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:48.326048 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf34c1a-954e-40ba-af90-af7f918b2ae3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:48.328131 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf34c1a-954e-40ba-af90-af7f918b2ae3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:48.350478 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a85d204-9ade-48b7-b06f-3e16f70a5d9b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:48.350478 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a85d204-9ade-48b7-b06f-3e16f70a5d9b service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:48.361021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 814/1618] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:45:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:45:48.361770 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a85d204-9ade-48b7-b06f-3e16f70a5d9b service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:48.372557 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a85d204-9ade-48b7-b06f-3e16f70a5d9b service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:48.379235 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c746aba7-9125-4615-a394-b8cfb464004f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:48.379907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c746aba7-9125-4615-a394-b8cfb464004f admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:48.380255 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c746aba7-9125-4615-a394-b8cfb464004f admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:48.380601 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c746aba7-9125-4615-a394-b8cfb464004f admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:48.381302 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c746aba7-9125-4615-a394-b8cfb464004f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:48.383834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c746aba7-9125-4615-a394-b8cfb464004f admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:48.475155 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 805/1619] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:45:48 2022] POST /identity/v3/projects => generated 379 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:48.475155 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf34c1a-954e-40ba-af90-af7f918b2ae3 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:48.496536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 815/1620] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:45:48 2022] POST /identity/v3/users => generated 288 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:48.497557 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c746aba7-9125-4615-a394-b8cfb464004f admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:48.515372 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c746aba7-9125-4615-a394-b8cfb464004f admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:48.564576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:48.578762 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf34c1a-954e-40ba-af90-af7f918b2ae3 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:48.585212 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24de634e-017c-4e9d-a2c4-62fa81dc964e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:48.586231 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24de634e-017c-4e9d-a2c4-62fa81dc964e admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:48.586477 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24de634e-017c-4e9d-a2c4-62fa81dc964e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:48.586773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24de634e-017c-4e9d-a2c4-62fa81dc964e admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:48.591890 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24de634e-017c-4e9d-a2c4-62fa81dc964e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:48.591890 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24de634e-017c-4e9d-a2c4-62fa81dc964e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:48.606987 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab5f1df-b6a9-428b-850b-c86eb82072d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:48.607763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab5f1df-b6a9-428b-850b-c86eb82072d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:48.608422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab5f1df-b6a9-428b-850b-c86eb82072d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:48.608422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab5f1df-b6a9-428b-850b-c86eb82072d7 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:48.610436 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab5f1df-b6a9-428b-850b-c86eb82072d7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:48.611666 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab5f1df-b6a9-428b-850b-c86eb82072d7 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:48.670716 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:48.674585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 806/1621] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:45:48 2022] POST /identity/v3/users => generated 290 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:48.677427 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24de634e-017c-4e9d-a2c4-62fa81dc964e admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:48.685510 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24de634e-017c-4e9d-a2c4-62fa81dc964e admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:48.691076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54aa13cf-57a6-4b5c-bc86-f197f5c6244f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:48.691348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54aa13cf-57a6-4b5c-bc86-f197f5c6244f admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:48.691629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54aa13cf-57a6-4b5c-bc86-f197f5c6244f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:48.691896 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54aa13cf-57a6-4b5c-bc86-f197f5c6244f admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:48.692572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54aa13cf-57a6-4b5c-bc86-f197f5c6244f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:48.694911 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54aa13cf-57a6-4b5c-bc86-f197f5c6244f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:48.710224 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 816/1622] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:45:48 2022] POST /identity/v3/users => generated 293 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:48.711958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab5f1df-b6a9-428b-850b-c86eb82072d7 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:48.724120 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab5f1df-b6a9-428b-850b-c86eb82072d7 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:48.728930 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216198fd-b680-42c0-9545-af7abc3b2d1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:48.729537 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216198fd-b680-42c0-9545-af7abc3b2d1e admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:48.729816 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216198fd-b680-42c0-9545-af7abc3b2d1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:48.730104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216198fd-b680-42c0-9545-af7abc3b2d1e admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:48.730900 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216198fd-b680-42c0-9545-af7abc3b2d1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:48.734880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216198fd-b680-42c0-9545-af7abc3b2d1e admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:48.742830 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 817/1623] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:48 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:48.743629 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216198fd-b680-42c0-9545-af7abc3b2d1e admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:48.753801 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216198fd-b680-42c0-9545-af7abc3b2d1e admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:48.758818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c204d8-4d41-44fd-aa72-7364fb271280 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:48.763516 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c204d8-4d41-44fd-aa72-7364fb271280 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:48.763516 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c204d8-4d41-44fd-aa72-7364fb271280 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:48.763516 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c204d8-4d41-44fd-aa72-7364fb271280 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:48.763516 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c204d8-4d41-44fd-aa72-7364fb271280 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:48.763516 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c204d8-4d41-44fd-aa72-7364fb271280 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:48.772754 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 818/1624] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:48 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:48.773550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c204d8-4d41-44fd-aa72-7364fb271280 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:48.783308 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c204d8-4d41-44fd-aa72-7364fb271280 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:48.787945 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4551338a-3018-46fe-bc39-cb722dc19546 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:48.788556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4551338a-3018-46fe-bc39-cb722dc19546 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:48.788834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4551338a-3018-46fe-bc39-cb722dc19546 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:48.789104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4551338a-3018-46fe-bc39-cb722dc19546 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:48.789742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4551338a-3018-46fe-bc39-cb722dc19546 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:48.791401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4551338a-3018-46fe-bc39-cb722dc19546 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:48.805992 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 807/1625] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:45:48 2022] POST /identity/v3/users => generated 296 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:45:48.810882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54aa13cf-57a6-4b5c-bc86-f197f5c6244f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:48.814233 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 819/1626] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:48 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:48.815320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4551338a-3018-46fe-bc39-cb722dc19546 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:48.827207 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54aa13cf-57a6-4b5c-bc86-f197f5c6244f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:48.828188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4551338a-3018-46fe-bc39-cb722dc19546 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:48.836547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac9dcbb-ca01-4c32-a418-7c3be011fa1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:48.838773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac9dcbb-ca01-4c32-a418-7c3be011fa1b admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:48.839901 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670809ad-73fa-49a5-b809-3e510a34c661 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:48.840436 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac9dcbb-ca01-4c32-a418-7c3be011fa1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:48.840722 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac9dcbb-ca01-4c32-a418-7c3be011fa1b admin admin] PATH_INFO: `/v3/projects/09d3e6a988c14b7e860b96d58901271a/users/5321878d963143abb3660600713115ea/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:48.841990 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670809ad-73fa-49a5-b809-3e510a34c661 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:48.842272 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670809ad-73fa-49a5-b809-3e510a34c661 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:48.842924 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670809ad-73fa-49a5-b809-3e510a34c661 admin admin] PATH_INFO: `/v3/projects/8ead447fbc024c66b835a0e1af4e746e/users/200bc2f8aa4c4527a6c812c5836db2c5/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:48.878501 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670809ad-73fa-49a5-b809-3e510a34c661 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ead447fbc024c66b835a0e1af4e746e, user_id=200bc2f8aa4c4527a6c812c5836db2c5, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:48.879655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac9dcbb-ca01-4c32-a418-7c3be011fa1b admin admin] RBAC: Authorizing `identity:create_grant(project_id=09d3e6a988c14b7e860b96d58901271a, user_id=5321878d963143abb3660600713115ea, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:48.882784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac9dcbb-ca01-4c32-a418-7c3be011fa1b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:48.883244 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670809ad-73fa-49a5-b809-3e510a34c661 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:48.976621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 808/1627] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:45:48 2022] PUT /identity/v3/projects/09d3e6a988c14b7e860b96d58901271a/users/5321878d963143abb3660600713115ea/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:45:48.982385 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac9dcbb-ca01-4c32-a418-7c3be011fa1b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:49.007530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 820/1628] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:45:48 2022] PUT /identity/v3/projects/8ead447fbc024c66b835a0e1af4e746e/users/200bc2f8aa4c4527a6c812c5836db2c5/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:45:49.008640 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670809ad-73fa-49a5-b809-3e510a34c661 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:49.183119 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac9dcbb-ca01-4c32-a418-7c3be011fa1b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:49.191043 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670809ad-73fa-49a5-b809-3e510a34c661 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:49.191043 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c459d84-838a-44d0-9f3c-8f019c18b070 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:49.191043 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c459d84-838a-44d0-9f3c-8f019c18b070 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:49.191043 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c459d84-838a-44d0-9f3c-8f019c18b070 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:49.191550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c459d84-838a-44d0-9f3c-8f019c18b070 admin admin] PATH_INFO: `/v3/projects/32153e84064043b8aaff202896b8d86c/users/106c3ba56cd84155bedc474fd2479355/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:49.191550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c459d84-838a-44d0-9f3c-8f019c18b070 admin admin] RBAC: Authorizing `identity:create_grant(project_id=32153e84064043b8aaff202896b8d86c, user_id=106c3ba56cd84155bedc474fd2479355, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:49.191550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c459d84-838a-44d0-9f3c-8f019c18b070 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:49.192878 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5254628e-dd2c-4d65-b75b-e3b1019a20dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:49.192878 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5254628e-dd2c-4d65-b75b-e3b1019a20dc admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:49.192878 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5254628e-dd2c-4d65-b75b-e3b1019a20dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:49.192878 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5254628e-dd2c-4d65-b75b-e3b1019a20dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:49.193235 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5254628e-dd2c-4d65-b75b-e3b1019a20dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:49.193235 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5254628e-dd2c-4d65-b75b-e3b1019a20dc admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:49.203994 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 809/1629] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:48 2022] GET /identity/v3/roles => generated 1420 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:49.208989 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5254628e-dd2c-4d65-b75b-e3b1019a20dc admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:49.219221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5254628e-dd2c-4d65-b75b-e3b1019a20dc admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:49.246199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fd420e-39bf-4ad7-8f1d-902ad416f363 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:49.246854 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fd420e-39bf-4ad7-8f1d-902ad416f363 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:49.251245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fd420e-39bf-4ad7-8f1d-902ad416f363 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:49.251245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fd420e-39bf-4ad7-8f1d-902ad416f363 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:49.251245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fd420e-39bf-4ad7-8f1d-902ad416f363 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:49.252335 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fd420e-39bf-4ad7-8f1d-902ad416f363 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:49.264941 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 810/1630] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:49 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:49.265637 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fd420e-39bf-4ad7-8f1d-902ad416f363 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:49.281354 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fd420e-39bf-4ad7-8f1d-902ad416f363 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:49.283662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 821/1631] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:45:49 2022] PUT /identity/v3/projects/32153e84064043b8aaff202896b8d86c/users/106c3ba56cd84155bedc474fd2479355/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:45:49.284637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c459d84-838a-44d0-9f3c-8f019c18b070 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:49.341556 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:49.379474 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0501e63f-44ba-4c8c-99ae-f0f1aeb35cf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:49.381028 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0501e63f-44ba-4c8c-99ae-f0f1aeb35cf4 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:49.381384 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0501e63f-44ba-4c8c-99ae-f0f1aeb35cf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:49.381780 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0501e63f-44ba-4c8c-99ae-f0f1aeb35cf4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:49.382676 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0501e63f-44ba-4c8c-99ae-f0f1aeb35cf4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:49.385348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0501e63f-44ba-4c8c-99ae-f0f1aeb35cf4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:49.406005 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 811/1632] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:49 2022] GET /identity/v3/roles => generated 1420 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:49.407633 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0501e63f-44ba-4c8c-99ae-f0f1aeb35cf4 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:49.410791 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c459d84-838a-44d0-9f3c-8f019c18b070 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:49.416213 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0b87aa-16af-48c8-a5b0-94c985042716 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:49.417150 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0b87aa-16af-48c8-a5b0-94c985042716 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:49.417611 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0b87aa-16af-48c8-a5b0-94c985042716 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:49.418143 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0b87aa-16af-48c8-a5b0-94c985042716 admin admin] PATH_INFO: `/v3/projects/dc3720bb351f43fc9ea163c42427bad6/users/6c58c3b8a7aa46aea41571aff63590b3/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:49.419328 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0501e63f-44ba-4c8c-99ae-f0f1aeb35cf4 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:49.431315 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c00d391-d235-44a3-868f-bc9790ef63e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:49.433077 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c00d391-d235-44a3-868f-bc9790ef63e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:49.433187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c00d391-d235-44a3-868f-bc9790ef63e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:49.433492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c00d391-d235-44a3-868f-bc9790ef63e3 admin admin] PATH_INFO: `/v3/projects/09d3e6a988c14b7e860b96d58901271a/users/5321878d963143abb3660600713115ea/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:49.437051 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c00d391-d235-44a3-868f-bc9790ef63e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=09d3e6a988c14b7e860b96d58901271a, user_id=5321878d963143abb3660600713115ea, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:49.440361 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c00d391-d235-44a3-868f-bc9790ef63e3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:49.444902 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0b87aa-16af-48c8-a5b0-94c985042716 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc3720bb351f43fc9ea163c42427bad6, user_id=6c58c3b8a7aa46aea41571aff63590b3, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:49.447984 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0b87aa-16af-48c8-a5b0-94c985042716 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:49.455994 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 812/1633] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:45:49 2022] PUT /identity/v3/projects/09d3e6a988c14b7e860b96d58901271a/users/5321878d963143abb3660600713115ea/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:45:49.459448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c00d391-d235-44a3-868f-bc9790ef63e3 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:49.490520 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:49.525944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 822/1634] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:45:49 2022] PUT /identity/v3/projects/dc3720bb351f43fc9ea163c42427bad6/users/6c58c3b8a7aa46aea41571aff63590b3/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:45:49.530983 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0b87aa-16af-48c8-a5b0-94c985042716 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:49.554033 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c00d391-d235-44a3-868f-bc9790ef63e3 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:49.655701 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0b87aa-16af-48c8-a5b0-94c985042716 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:49.663120 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f89843-8a4e-4886-82db-2a2e045eb41d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:49.669322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f89843-8a4e-4886-82db-2a2e045eb41d admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:49.670299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f89843-8a4e-4886-82db-2a2e045eb41d admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:49.670743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f89843-8a4e-4886-82db-2a2e045eb41d admin admin] PATH_INFO: `/v3/projects/8ead447fbc024c66b835a0e1af4e746e/users/200bc2f8aa4c4527a6c812c5836db2c5/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:49.674923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f89843-8a4e-4886-82db-2a2e045eb41d admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ead447fbc024c66b835a0e1af4e746e, user_id=200bc2f8aa4c4527a6c812c5836db2c5, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:49.680922 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f89843-8a4e-4886-82db-2a2e045eb41d admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:49.684126 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff00347-ad19-4fa3-8bb5-0a789ad9d924 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:49.685289 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff00347-ad19-4fa3-8bb5-0a789ad9d924 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:49.685570 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff00347-ad19-4fa3-8bb5-0a789ad9d924 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:49.685846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff00347-ad19-4fa3-8bb5-0a789ad9d924 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:49.686540 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff00347-ad19-4fa3-8bb5-0a789ad9d924 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:49.689538 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff00347-ad19-4fa3-8bb5-0a789ad9d924 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:49.714027 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 823/1635] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:45:49 2022] PUT /identity/v3/projects/8ead447fbc024c66b835a0e1af4e746e/users/200bc2f8aa4c4527a6c812c5836db2c5/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:45:49.714027 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fe11c7-c814-45e0-ab6b-321c5229c1c2 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:49.714027 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fe11c7-c814-45e0-ab6b-321c5229c1c2 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:49.714027 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fe11c7-c814-45e0-ab6b-321c5229c1c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:49.714634 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 813/1636] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:49 2022] GET /identity/v3/roles => generated 1420 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:49.714634 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff00347-ad19-4fa3-8bb5-0a789ad9d924 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:49.753999 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff00347-ad19-4fa3-8bb5-0a789ad9d924 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:49.757330 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-44fe11c7-c814-45e0-ab6b-321c5229c1c2 None None] MFA Rules not processed for user `5321878d963143abb3660600713115ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:49.758995 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-44fe11c7-c814-45e0-ab6b-321c5229c1c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:49.852158 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 824/1637] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:49 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:49.853607 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32afbfb4-1bc8-487d-afb6-968029d77a0f None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:49.853961 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32afbfb4-1bc8-487d-afb6-968029d77a0f None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:49.855009 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275221c9-33bf-4129-b5ea-74c97c53a766 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:49.855872 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275221c9-33bf-4129-b5ea-74c97c53a766 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:49.856445 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275221c9-33bf-4129-b5ea-74c97c53a766 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:49.856562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32afbfb4-1bc8-487d-afb6-968029d77a0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:49.856958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275221c9-33bf-4129-b5ea-74c97c53a766 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:49.857791 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275221c9-33bf-4129-b5ea-74c97c53a766 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:49.860311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275221c9-33bf-4129-b5ea-74c97c53a766 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:49.882452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 814/1638] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:45:49 2022] GET /identity/v3/roles => generated 1420 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:45:49.883225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275221c9-33bf-4129-b5ea-74c97c53a766 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:49.898313 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275221c9-33bf-4129-b5ea-74c97c53a766 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:49.903177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05cc855-b091-41b7-85d0-7db70cf34b4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:49.903841 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05cc855-b091-41b7-85d0-7db70cf34b4c admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:49.904523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05cc855-b091-41b7-85d0-7db70cf34b4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:49.904523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05cc855-b091-41b7-85d0-7db70cf34b4c admin admin] PATH_INFO: `/v3/projects/32153e84064043b8aaff202896b8d86c/users/106c3ba56cd84155bedc474fd2479355/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:49.906358 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05cc855-b091-41b7-85d0-7db70cf34b4c admin admin] RBAC: Authorizing `identity:create_grant(project_id=32153e84064043b8aaff202896b8d86c, user_id=106c3ba56cd84155bedc474fd2479355, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:49.910178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05cc855-b091-41b7-85d0-7db70cf34b4c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:49.913358 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-32afbfb4-1bc8-487d-afb6-968029d77a0f None None] MFA Rules not processed for user `200bc2f8aa4c4527a6c812c5836db2c5`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:49.916123 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-32afbfb4-1bc8-487d-afb6-968029d77a0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:49.935764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 815/1639] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:45:49 2022] PUT /identity/v3/projects/32153e84064043b8aaff202896b8d86c/users/106c3ba56cd84155bedc474fd2479355/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:45:49.946315 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05cc855-b091-41b7-85d0-7db70cf34b4c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:50.045300 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05cc855-b091-41b7-85d0-7db70cf34b4c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:50.050523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def40105-8f56-4e82-9b22-4fc8a88d4c19 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:50.051319 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def40105-8f56-4e82-9b22-4fc8a88d4c19 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:50.051810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def40105-8f56-4e82-9b22-4fc8a88d4c19 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:50.052181 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def40105-8f56-4e82-9b22-4fc8a88d4c19 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:50.326227 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 825/1640] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:49 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:50.326227 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32afbfb4-1bc8-487d-afb6-968029d77a0f None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:50.354482 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def40105-8f56-4e82-9b22-4fc8a88d4c19 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:50.354482 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def40105-8f56-4e82-9b22-4fc8a88d4c19 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:50.354482 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 816/1641] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:45:49 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 253 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:45:50.354482 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f86fbd-81b3-4264-8fc7-72d417ff4120 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:50.354482 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f86fbd-81b3-4264-8fc7-72d417ff4120 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:50.354482 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f86fbd-81b3-4264-8fc7-72d417ff4120 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:50.355410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32afbfb4-1bc8-487d-afb6-968029d77a0f None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:50.355410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a99a94e-4fd9-4460-af77-c3eecfe4cb7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:50.355410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a99a94e-4fd9-4460-af77-c3eecfe4cb7a admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:50.355410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a99a94e-4fd9-4460-af77-c3eecfe4cb7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:50.356103 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a99a94e-4fd9-4460-af77-c3eecfe4cb7a admin admin] PATH_INFO: `/v3/projects/dc3720bb351f43fc9ea163c42427bad6/users/6c58c3b8a7aa46aea41571aff63590b3/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:50.356103 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a99a94e-4fd9-4460-af77-c3eecfe4cb7a admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc3720bb351f43fc9ea163c42427bad6, user_id=6c58c3b8a7aa46aea41571aff63590b3, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:50.356103 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a99a94e-4fd9-4460-af77-c3eecfe4cb7a admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:50.356333 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-a9f86fbd-81b3-4264-8fc7-72d417ff4120 None None] MFA Rules not processed for user `106c3ba56cd84155bedc474fd2479355`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:50.356333 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a9f86fbd-81b3-4264-8fc7-72d417ff4120 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:50.356521 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 826/1642] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:45:50 2022] PUT /identity/v3/projects/dc3720bb351f43fc9ea163c42427bad6/users/6c58c3b8a7aa46aea41571aff63590b3/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:45:50.356521 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a99a94e-4fd9-4460-af77-c3eecfe4cb7a admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:50.356695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:50.356851 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a99a94e-4fd9-4460-af77-c3eecfe4cb7a admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:50.356851 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0f7cfd-1eac-419f-b166-5f0b3c68cea2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:50.356851 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0f7cfd-1eac-419f-b166-5f0b3c68cea2 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:50.356851 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0f7cfd-1eac-419f-b166-5f0b3c68cea2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:50.357132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0f7cfd-1eac-419f-b166-5f0b3c68cea2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:50.366398 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 817/1643] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:50 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:50.369448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26230a19-ff1f-4fdb-a72b-1c052ec82c82 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:50.370255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26230a19-ff1f-4fdb-a72b-1c052ec82c82 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:50.370255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26230a19-ff1f-4fdb-a72b-1c052ec82c82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:50.416189 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-26230a19-ff1f-4fdb-a72b-1c052ec82c82 None None] MFA Rules not processed for user `6c58c3b8a7aa46aea41571aff63590b3`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:50.418522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-26230a19-ff1f-4fdb-a72b-1c052ec82c82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:50.481758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0f7cfd-1eac-419f-b166-5f0b3c68cea2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:50.482190 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0f7cfd-1eac-419f-b166-5f0b3c68cea2 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:50.518144 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 827/1644] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:45:50 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 274 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:45:50.521021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:50.586333 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 818/1645] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:50 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:50.635643 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0f7cfd-1eac-419f-b166-5f0b3c68cea2 service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:50.646737 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0f7cfd-1eac-419f-b166-5f0b3c68cea2 service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:50.651199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2baba02a-84c6-4ebe-8528-1853f78b9a62 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:50.651999 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baba02a-84c6-4ebe-8528-1853f78b9a62 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:50.652331 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baba02a-84c6-4ebe-8528-1853f78b9a62 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:50.652608 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baba02a-84c6-4ebe-8528-1853f78b9a62 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:50.665838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2baba02a-84c6-4ebe-8528-1853f78b9a62 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:50.667453 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2baba02a-84c6-4ebe-8528-1853f78b9a62 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:50.711973 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 828/1646] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:45:50 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:45:50.763469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26230a19-ff1f-4fdb-a72b-1c052ec82c82 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:50.780391 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26230a19-ff1f-4fdb-a72b-1c052ec82c82 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:50.791217 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8d0888-03d9-4da0-b7af-3ddbdc7fdb3a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:50.792441 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8d0888-03d9-4da0-b7af-3ddbdc7fdb3a service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:50.792770 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8d0888-03d9-4da0-b7af-3ddbdc7fdb3a service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:50.793223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8d0888-03d9-4da0-b7af-3ddbdc7fdb3a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:50.803521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8d0888-03d9-4da0-b7af-3ddbdc7fdb3a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:50.804964 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8d0888-03d9-4da0-b7af-3ddbdc7fdb3a service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:50.833612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 819/1647] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:45:50 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 71 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:45:51.045113 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ec2663-7804-4e15-98ca-67a6a989c937 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:51.045433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ec2663-7804-4e15-98ca-67a6a989c937 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:51.046101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ec2663-7804-4e15-98ca-67a6a989c937 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:51.048342 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 829/1648] 10.222.0.39 () {54 vars in 961 bytes} [Fri Sep 30 09:45:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:45:51.065209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f00e7b-df58-427a-8a6d-972d7d295e56 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:51.065504 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f00e7b-df58-427a-8a6d-972d7d295e56 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:51.065784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f00e7b-df58-427a-8a6d-972d7d295e56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:51.163551 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e9f00e7b-df58-427a-8a6d-972d7d295e56 None None] MFA Rules not processed for user `7ff68fb87700423ab53c90520a6bd3b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:51.165419 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e9f00e7b-df58-427a-8a6d-972d7d295e56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:51.252849 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ec2663-7804-4e15-98ca-67a6a989c937 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:51.275349 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ec2663-7804-4e15-98ca-67a6a989c937 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:51.279646 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9a0684-9cd4-445b-b48b-2cdde4b785f9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:51.281359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9a0684-9cd4-445b-b48b-2cdde4b785f9 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:51.281670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9a0684-9cd4-445b-b48b-2cdde4b785f9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:51.282546 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9a0684-9cd4-445b-b48b-2cdde4b785f9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:51.291077 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 820/1649] 10.222.0.39 () {58 vars in 1058 bytes} [Fri Sep 30 09:45:51 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:51.299552 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1092f2d7-acc4-45cf-9f5b-bbbf7b0e1902 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:51.299841 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1092f2d7-acc4-45cf-9f5b-bbbf7b0e1902 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:51.305529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1092f2d7-acc4-45cf-9f5b-bbbf7b0e1902 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:51.306418 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 821/1650] 10.222.0.39 () {54 vars in 912 bytes} [Fri Sep 30 09:45:51 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:45:51.307888 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e514828f-6ca7-4b05-9131-cba54314f8eb None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:51.308214 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e514828f-6ca7-4b05-9131-cba54314f8eb None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:51.308493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e514828f-6ca7-4b05-9131-cba54314f8eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:51.335504 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e514828f-6ca7-4b05-9131-cba54314f8eb None None] MFA Rules not processed for user `7ff68fb87700423ab53c90520a6bd3b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:51.337339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e514828f-6ca7-4b05-9131-cba54314f8eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:51.353145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 822/1651] 10.222.0.39 () {58 vars in 1009 bytes} [Fri Sep 30 09:45:51 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:51.374391 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e514828f-6ca7-4b05-9131-cba54314f8eb None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:51.386764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e514828f-6ca7-4b05-9131-cba54314f8eb None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:51.394825 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4235027-effb-4383-9b95-392824be5b96 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:51.396084 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4235027-effb-4383-9b95-392824be5b96 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:51.398893 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4235027-effb-4383-9b95-392824be5b96 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:51.399173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4235027-effb-4383-9b95-392824be5b96 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:51.407285 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9a0684-9cd4-445b-b48b-2cdde4b785f9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:51.408751 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9a0684-9cd4-445b-b48b-2cdde4b785f9 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:51.424396 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 830/1652] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:45:51 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:45:51.427942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:51.486249 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4235027-effb-4383-9b95-392824be5b96 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:51.488063 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4235027-effb-4383-9b95-392824be5b96 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:51.501129 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 823/1653] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:45:51 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:45:51.513828 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:52.210858 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9a0684-9cd4-445b-b48b-2cdde4b785f9 service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:52.223769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9a0684-9cd4-445b-b48b-2cdde4b785f9 service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:52.233540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0a31fa-a2a2-4355-81fb-7e60b97e8690 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:52.234391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0a31fa-a2a2-4355-81fb-7e60b97e8690 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:52.235552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0a31fa-a2a2-4355-81fb-7e60b97e8690 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:52.236015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0a31fa-a2a2-4355-81fb-7e60b97e8690 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:52.265496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0a31fa-a2a2-4355-81fb-7e60b97e8690 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:52.267869 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0a31fa-a2a2-4355-81fb-7e60b97e8690 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:52.293530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 831/1654] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:45:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:45:52.983480 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:55.755457 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca0cce6-9d58-4012-bbc0-116d4a45ca08 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:55.756472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca0cce6-9d58-4012-bbc0-116d4a45ca08 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:55.756884 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca0cce6-9d58-4012-bbc0-116d4a45ca08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:55.796832 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94a12ab-1047-4094-9767-d6692c536933 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:55.797410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94a12ab-1047-4094-9767-d6692c536933 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:55.797410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94a12ab-1047-4094-9767-d6692c536933 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:55.798781 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 832/1655] 10.222.0.39 () {54 vars in 961 bytes} [Fri Sep 30 09:45:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:45:55.807305 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e37529-9fee-4da0-b6c7-d115cca3f4b3 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:55.807581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e37529-9fee-4da0-b6c7-d115cca3f4b3 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:55.807857 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e37529-9fee-4da0-b6c7-d115cca3f4b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:55.843322 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-0ca0cce6-9d58-4012-bbc0-116d4a45ca08 None None] MFA Rules not processed for user `7ad21da05fc246d489b0afa641f20216`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:55.845972 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0ca0cce6-9d58-4012-bbc0-116d4a45ca08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:55.856493 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-a6e37529-9fee-4da0-b6c7-d115cca3f4b3 None None] MFA Rules not processed for user `7ff68fb87700423ab53c90520a6bd3b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:55.857984 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a6e37529-9fee-4da0-b6c7-d115cca3f4b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:55.891020 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 833/1656] 10.222.0.39 () {58 vars in 1058 bytes} [Fri Sep 30 09:45:55 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:45:55.897698 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9062433-0ba7-43eb-9a56-9f07bd0464e0 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:55.898748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9062433-0ba7-43eb-9a56-9f07bd0464e0 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:55.899236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9062433-0ba7-43eb-9a56-9f07bd0464e0 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:55.901095 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 834/1657] 10.222.0.39 () {54 vars in 912 bytes} [Fri Sep 30 09:45:55 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:45:55.907508 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6043526d-12b3-4f93-8990-1ede9eba1627 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:55.907928 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6043526d-12b3-4f93-8990-1ede9eba1627 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:55.908355 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6043526d-12b3-4f93-8990-1ede9eba1627 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:55.977757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-6043526d-12b3-4f93-8990-1ede9eba1627 None None] MFA Rules not processed for user `7ff68fb87700423ab53c90520a6bd3b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:55.981494 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-6043526d-12b3-4f93-8990-1ede9eba1627 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:56.006189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 835/1658] 10.222.0.39 () {58 vars in 1009 bytes} [Fri Sep 30 09:45:55 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:45:56.035093 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6043526d-12b3-4f93-8990-1ede9eba1627 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:56.047815 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 824/1659] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:55 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 285 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:56.051760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca0cce6-9d58-4012-bbc0-116d4a45ca08 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:56.086397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca0cce6-9d58-4012-bbc0-116d4a45ca08 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:56.100869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d91edd-2fdc-404d-9807-61e626aecb47 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:56.101208 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d91edd-2fdc-404d-9807-61e626aecb47 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:56.101501 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d91edd-2fdc-404d-9807-61e626aecb47 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:56.102214 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d91edd-2fdc-404d-9807-61e626aecb47 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:56.133107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d91edd-2fdc-404d-9807-61e626aecb47 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:56.135737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d91edd-2fdc-404d-9807-61e626aecb47 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:56.168475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 825/1660] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:45:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:45:56.172489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a63040-51a4-44e5-936e-7d06a01f7c01 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:56.172791 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a63040-51a4-44e5-936e-7d06a01f7c01 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:56.178264 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a63040-51a4-44e5-936e-7d06a01f7c01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:56.233621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6043526d-12b3-4f93-8990-1ede9eba1627 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:56.267448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-50a63040-51a4-44e5-936e-7d06a01f7c01 None None] MFA Rules not processed for user `7ad21da05fc246d489b0afa641f20216`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:56.271490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90590292-d0fd-4f73-97b3-fa73fdd5297e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:56.274227 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90590292-d0fd-4f73-97b3-fa73fdd5297e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:56.274227 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90590292-d0fd-4f73-97b3-fa73fdd5297e admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:56.274432 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-50a63040-51a4-44e5-936e-7d06a01f7c01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:56.274523 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90590292-d0fd-4f73-97b3-fa73fdd5297e admin admin] PATH_INFO: `/v3/users/76092190b13945b284c9bc6005bc5319` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:56.289176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90590292-d0fd-4f73-97b3-fa73fdd5297e admin admin] RBAC: Authorizing `identity:delete_user(user_id=76092190b13945b284c9bc6005bc5319)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:56.293832 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: WARNING py.warnings [None req-90590292-d0fd-4f73-97b3-fa73fdd5297e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 30 09:45:56.293832 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: warnings.warn(msg) Sep 30 09:45:56.293832 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: Sep 30 09:45:56.297955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90590292-d0fd-4f73-97b3-fa73fdd5297e admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:56.337446 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 826/1661] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:56 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:56.355769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a63040-51a4-44e5-936e-7d06a01f7c01 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:56.380975 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a63040-51a4-44e5-936e-7d06a01f7c01 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:56.421763 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f909243-b900-41ac-8ad7-3fc91fe0fd88 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:56.422613 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f909243-b900-41ac-8ad7-3fc91fe0fd88 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:56.423229 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f909243-b900-41ac-8ad7-3fc91fe0fd88 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:56.425352 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f909243-b900-41ac-8ad7-3fc91fe0fd88 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:56.470053 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f909243-b900-41ac-8ad7-3fc91fe0fd88 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:56.471937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f909243-b900-41ac-8ad7-3fc91fe0fd88 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:56.496375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 827/1662] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:45:56 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:45:56.650052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-90590292-d0fd-4f73-97b3-fa73fdd5297e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '76092190b13945b284c9bc6005bc5319'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:45:56.714695 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-90590292-d0fd-4f73-97b3-fa73fdd5297e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '76092190b13945b284c9bc6005bc5319'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:45:56.716142 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-90590292-d0fd-4f73-97b3-fa73fdd5297e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '76092190b13945b284c9bc6005bc5319'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:45:56.822137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-90590292-d0fd-4f73-97b3-fa73fdd5297e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '76092190b13945b284c9bc6005bc5319'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:45:56.852640 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-90590292-d0fd-4f73-97b3-fa73fdd5297e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '76092190b13945b284c9bc6005bc5319'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:45:56.872330 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 836/1663] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:45:56 2022] DELETE /identity/v3/users/76092190b13945b284c9bc6005bc5319 => generated 0 bytes in 838 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:45:56.877327 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840e4e13-c87f-452b-9338-380d38e16e8b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:56.877912 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840e4e13-c87f-452b-9338-380d38e16e8b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:56.877912 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840e4e13-c87f-452b-9338-380d38e16e8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:56.919308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-840e4e13-c87f-452b-9338-380d38e16e8b None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:56.920486 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-840e4e13-c87f-452b-9338-380d38e16e8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:57.064255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 828/1664] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:57.109066 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90590292-d0fd-4f73-97b3-fa73fdd5297e admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:57.111542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-90590292-d0fd-4f73-97b3-fa73fdd5297e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:57.203697 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:57.259338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90590292-d0fd-4f73-97b3-fa73fdd5297e admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:57.264404 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6ecfbe-6ee7-4718-b3ec-70ae64644417 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:57.267778 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6ecfbe-6ee7-4718-b3ec-70ae64644417 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:57.268220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6ecfbe-6ee7-4718-b3ec-70ae64644417 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:57.268646 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6ecfbe-6ee7-4718-b3ec-70ae64644417 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:57.294651 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6ecfbe-6ee7-4718-b3ec-70ae64644417 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:57.301626 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6ecfbe-6ee7-4718-b3ec-70ae64644417 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:57.324570 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 837/1665] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:45:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:45:57.873896 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a5500a-5547-40f1-bd91-4ec2cac3a82f None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:57.874195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a5500a-5547-40f1-bd91-4ec2cac3a82f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:57.874477 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a5500a-5547-40f1-bd91-4ec2cac3a82f None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:57.878614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 829/1666] 10.222.0.39 () {54 vars in 912 bytes} [Fri Sep 30 09:45:57 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:45:57.883915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45afc85c-c6ee-4957-8bd7-39fbaec29d62 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:57.884366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45afc85c-c6ee-4957-8bd7-39fbaec29d62 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:57.884505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45afc85c-c6ee-4957-8bd7-39fbaec29d62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:57.918753 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-45afc85c-c6ee-4957-8bd7-39fbaec29d62 None None] MFA Rules not processed for user `9266babd21414ae9998716e5a0478e9d`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:57.920343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-45afc85c-c6ee-4957-8bd7-39fbaec29d62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:57.945390 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 838/1667] 10.222.0.39 () {58 vars in 1009 bytes} [Fri Sep 30 09:45:57 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:57.957942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a5500a-5547-40f1-bd91-4ec2cac3a82f None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:57.959822 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-63a5500a-5547-40f1-bd91-4ec2cac3a82f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:57.976909 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a5500a-5547-40f1-bd91-4ec2cac3a82f None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:57.981480 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a013d4c-7750-4a52-903b-b37fb326bf98 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:57.982221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a013d4c-7750-4a52-903b-b37fb326bf98 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:57.982622 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a013d4c-7750-4a52-903b-b37fb326bf98 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:57.983233 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a013d4c-7750-4a52-903b-b37fb326bf98 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:57.993230 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a013d4c-7750-4a52-903b-b37fb326bf98 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:57.995666 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a013d4c-7750-4a52-903b-b37fb326bf98 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:58.010264 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 830/1668] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:45:57 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:45:59.037408 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45afc85c-c6ee-4957-8bd7-39fbaec29d62 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:59.039148 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-45afc85c-c6ee-4957-8bd7-39fbaec29d62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:59.085694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45afc85c-c6ee-4957-8bd7-39fbaec29d62 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:59.101287 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0b4888-66db-4215-ae54-d13d0a0ff45f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:59.108371 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0b4888-66db-4215-ae54-d13d0a0ff45f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:59.108371 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0b4888-66db-4215-ae54-d13d0a0ff45f admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:59.108371 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0b4888-66db-4215-ae54-d13d0a0ff45f admin admin] PATH_INFO: `/v3/projects/66fd4af7b1ba4f02bd6525daa760b783` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:59.111152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0b4888-66db-4215-ae54-d13d0a0ff45f admin admin] RBAC: Authorizing `identity:delete_project(project_id=66fd4af7b1ba4f02bd6525daa760b783)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:59.114187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0b4888-66db-4215-ae54-d13d0a0ff45f admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:59.175067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a013d4c-7750-4a52-903b-b37fb326bf98 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:59.183917 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5a013d4c-7750-4a52-903b-b37fb326bf98 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:59.336800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:45:59.378951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-9a0b4888-66db-4215-ae54-d13d0a0ff45f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '66fd4af7b1ba4f02bd6525daa760b783', 'request_id': 'req-9a0b4888-66db-4215-ae54-d13d0a0ff45f'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:45:59.401386 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-9a0b4888-66db-4215-ae54-d13d0a0ff45f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '66fd4af7b1ba4f02bd6525daa760b783', 'request_id': 'req-9a0b4888-66db-4215-ae54-d13d0a0ff45f'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:45:59.421683 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-9a0b4888-66db-4215-ae54-d13d0a0ff45f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '66fd4af7b1ba4f02bd6525daa760b783', 'request_id': 'req-9a0b4888-66db-4215-ae54-d13d0a0ff45f'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:45:59.438900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a013d4c-7750-4a52-903b-b37fb326bf98 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:59.528495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-9a0b4888-66db-4215-ae54-d13d0a0ff45f admin admin] The token cache is being invalidate because project 66fd4af7b1ba4f02bd6525daa760b783 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:45:59.528495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-9a0b4888-66db-4215-ae54-d13d0a0ff45f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:45:59.528495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-9a0b4888-66db-4215-ae54-d13d0a0ff45f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:45:59.533023 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 839/1669] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:45:59 2022] DELETE /identity/v3/projects/66fd4af7b1ba4f02bd6525daa760b783 => generated 0 bytes in 493 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:45:59.534716 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d1c47d-9643-4a83-a4a0-20bf3b29e7dd None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:59.535174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d1c47d-9643-4a83-a4a0-20bf3b29e7dd None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:59.535587 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d1c47d-9643-4a83-a4a0-20bf3b29e7dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:59.585350 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-39d1c47d-9643-4a83-a4a0-20bf3b29e7dd None None] MFA Rules not processed for user `7ad21da05fc246d489b0afa641f20216`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:59.586801 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-39d1c47d-9643-4a83-a4a0-20bf3b29e7dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:59.604608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fd7f1f-d6ce-4527-b873-b73c2737ce82 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '49a3011c103b4e48904f876f81d27fba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:59.605218 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fd7f1f-d6ce-4527-b873-b73c2737ce82 service placement] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:59.605497 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fd7f1f-d6ce-4527-b873-b73c2737ce82 service placement] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:59.605785 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fd7f1f-d6ce-4527-b873-b73c2737ce82 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:59.608832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b2fd7f1f-d6ce-4527-b873-b73c2737ce82 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:59.626051 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2fd7f1f-d6ce-4527-b873-b73c2737ce82 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:59.627787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2fd7f1f-d6ce-4527-b873-b73c2737ce82 service placement] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:59.640277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 831/1670] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:45:59 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 466 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:45:59.641462 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fd7f1f-d6ce-4527-b873-b73c2737ce82 service placement] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:45:59.643187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b2fd7f1f-d6ce-4527-b873-b73c2737ce82 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:59.724856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 840/1671] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:59 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:45:59.726211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f0723a-f511-4218-a32b-f1448ce0fc0f None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:59.726502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f0723a-f511-4218-a32b-f1448ce0fc0f None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:59.726841 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f0723a-f511-4218-a32b-f1448ce0fc0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:59.765656 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fd7f1f-d6ce-4527-b873-b73c2737ce82 service placement] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:45:59.787039 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14a086b-fc31-4496-93bf-cab76882b572 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:45:59.787696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14a086b-fc31-4496-93bf-cab76882b572 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:45:59.788004 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14a086b-fc31-4496-93bf-cab76882b572 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:45:59.788255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14a086b-fc31-4496-93bf-cab76882b572 admin admin] PATH_INFO: `/v3/users/5321878d963143abb3660600713115ea` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:45:59.789751 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14a086b-fc31-4496-93bf-cab76882b572 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5321878d963143abb3660600713115ea)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:45:59.794080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14a086b-fc31-4496-93bf-cab76882b572 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:45:59.797126 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-47f0723a-f511-4218-a32b-f1448ce0fc0f None None] MFA Rules not processed for user `ae7915e1b2474d9384a0fdcb2ea177b6`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:45:59.798353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-47f0723a-f511-4218-a32b-f1448ce0fc0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:45:59.917013 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:00.016391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 841/1672] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:45:59 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 291 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:00.038338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85a3a87-9550-4d4a-b644-64a3af243120 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:00.042051 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85a3a87-9550-4d4a-b644-64a3af243120 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:00.042051 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85a3a87-9550-4d4a-b644-64a3af243120 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:00.082581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-a85a3a87-9550-4d4a-b644-64a3af243120 None None] MFA Rules not processed for user `ae7915e1b2474d9384a0fdcb2ea177b6`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:00.083918 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a85a3a87-9550-4d4a-b644-64a3af243120 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:00.135440 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 842/1673] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:00 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:00.136555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f14a086b-fc31-4496-93bf-cab76882b572 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5321878d963143abb3660600713115ea'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:00.142879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f14a086b-fc31-4496-93bf-cab76882b572 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5321878d963143abb3660600713115ea'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:00.147654 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85a3a87-9550-4d4a-b644-64a3af243120 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:00.148552 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f14a086b-fc31-4496-93bf-cab76882b572 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5321878d963143abb3660600713115ea'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:00.150968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a85a3a87-9550-4d4a-b644-64a3af243120 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:00.181070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f14a086b-fc31-4496-93bf-cab76882b572 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5321878d963143abb3660600713115ea'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:00.277463 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f14a086b-fc31-4496-93bf-cab76882b572 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5321878d963143abb3660600713115ea'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:00.296673 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:00.342929 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 832/1674] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:45:59 2022] DELETE /identity/v3/users/5321878d963143abb3660600713115ea => generated 0 bytes in 702 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:00.388725 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85a3a87-9550-4d4a-b644-64a3af243120 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:00.545389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14a086b-fc31-4496-93bf-cab76882b572 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:00.546942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f14a086b-fc31-4496-93bf-cab76882b572 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:00.587292 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc5f84b-f4f3-4760-852b-41e4236ad8be service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:00.587926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc5f84b-f4f3-4760-852b-41e4236ad8be service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:00.588173 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc5f84b-f4f3-4760-852b-41e4236ad8be service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:00.588452 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc5f84b-f4f3-4760-852b-41e4236ad8be service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:00.596203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-2fc5f84b-f4f3-4760-852b-41e4236ad8be service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:00.751499 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14a086b-fc31-4496-93bf-cab76882b572 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:00.759598 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af2b4ad-f01b-49a8-8530-8c21562a2644 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:00.760255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af2b4ad-f01b-49a8-8530-8c21562a2644 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:00.760531 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af2b4ad-f01b-49a8-8530-8c21562a2644 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:00.760804 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af2b4ad-f01b-49a8-8530-8c21562a2644 admin admin] PATH_INFO: `/v3/users/f91b594c06c1445198c7285f6053a4f1` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:00.761964 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af2b4ad-f01b-49a8-8530-8c21562a2644 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f91b594c06c1445198c7285f6053a4f1)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:00.764242 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af2b4ad-f01b-49a8-8530-8c21562a2644 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:00.829475 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc5f84b-f4f3-4760-852b-41e4236ad8be service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:00.834621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc5f84b-f4f3-4760-852b-41e4236ad8be service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:00.850785 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 843/1675] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:00 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 704 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:46:01.057229 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0af2b4ad-f01b-49a8-8530-8c21562a2644 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f91b594c06c1445198c7285f6053a4f1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:01.058534 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0af2b4ad-f01b-49a8-8530-8c21562a2644 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f91b594c06c1445198c7285f6053a4f1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:01.062721 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0af2b4ad-f01b-49a8-8530-8c21562a2644 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f91b594c06c1445198c7285f6053a4f1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:01.086362 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0af2b4ad-f01b-49a8-8530-8c21562a2644 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f91b594c06c1445198c7285f6053a4f1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:01.208485 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0af2b4ad-f01b-49a8-8530-8c21562a2644 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f91b594c06c1445198c7285f6053a4f1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:01.291595 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 833/1676] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:46:00 2022] DELETE /identity/v3/users/f91b594c06c1445198c7285f6053a4f1 => generated 0 bytes in 747 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:01.297393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38f2c0b-3767-41f0-96cd-e085a81243e6 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:01.297885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38f2c0b-3767-41f0-96cd-e085a81243e6 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:01.298196 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38f2c0b-3767-41f0-96cd-e085a81243e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:01.364325 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-c38f2c0b-3767-41f0-96cd-e085a81243e6 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:01.367211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c38f2c0b-3767-41f0-96cd-e085a81243e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:01.582347 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 844/1677] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 286 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:01.628744 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af2b4ad-f01b-49a8-8530-8c21562a2644 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:01.630735 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0af2b4ad-f01b-49a8-8530-8c21562a2644 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:01.660979 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6572c255-b17f-4889-be4f-121de0a54f24 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:01.661628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6572c255-b17f-4889-be4f-121de0a54f24 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:01.662117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6572c255-b17f-4889-be4f-121de0a54f24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:01.727524 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-6572c255-b17f-4889-be4f-121de0a54f24 None None] MFA Rules not processed for user `8c4ced79c3ff4842b30b472acb555cd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:01.730267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-6572c255-b17f-4889-be4f-121de0a54f24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:01.796045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af2b4ad-f01b-49a8-8530-8c21562a2644 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:01.802324 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0f59c4-fc33-45cb-83c7-687f9b4c25b5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:01.803820 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0f59c4-fc33-45cb-83c7-687f9b4c25b5 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:01.804302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0f59c4-fc33-45cb-83c7-687f9b4c25b5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:01.804733 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0f59c4-fc33-45cb-83c7-687f9b4c25b5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:01.820593 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:01.830730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0f59c4-fc33-45cb-83c7-687f9b4c25b5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:01.838247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0f59c4-fc33-45cb-83c7-687f9b4c25b5 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:01.861223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 834/1678] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:46:01.894401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 845/1679] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:01 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:01.910523 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6572c255-b17f-4889-be4f-121de0a54f24 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:01.916217 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-6572c255-b17f-4889-be4f-121de0a54f24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:01.918118 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7460db-1f88-46b7-973c-b3ad62af9921 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:01.918418 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7460db-1f88-46b7-973c-b3ad62af9921 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:01.918794 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7460db-1f88-46b7-973c-b3ad62af9921 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:01.920167 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 835/1680] 10.222.0.39 () {54 vars in 912 bytes} [Fri Sep 30 09:46:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:46:01.928837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a50eb82-d4d5-4365-b6f5-2314f54d3f55 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:01.929363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a50eb82-d4d5-4365-b6f5-2314f54d3f55 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:01.929771 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a50eb82-d4d5-4365-b6f5-2314f54d3f55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:01.959318 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9a50eb82-d4d5-4365-b6f5-2314f54d3f55 None None] MFA Rules not processed for user `9266babd21414ae9998716e5a0478e9d`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:01.961467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9a50eb82-d4d5-4365-b6f5-2314f54d3f55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:01.995053 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 836/1681] 10.222.0.39 () {58 vars in 1009 bytes} [Fri Sep 30 09:46:01 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:01.996507 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a50eb82-d4d5-4365-b6f5-2314f54d3f55 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:02.005514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9a50eb82-d4d5-4365-b6f5-2314f54d3f55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:02.055702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a50eb82-d4d5-4365-b6f5-2314f54d3f55 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:02.071322 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f66306-d2f0-4a2f-a95e-90353044e231 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:02.072052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f66306-d2f0-4a2f-a95e-90353044e231 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:02.072333 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f66306-d2f0-4a2f-a95e-90353044e231 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:02.074666 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f66306-d2f0-4a2f-a95e-90353044e231 admin admin] PATH_INFO: `/v3/projects/09d3e6a988c14b7e860b96d58901271a` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:02.076480 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f66306-d2f0-4a2f-a95e-90353044e231 admin admin] RBAC: Authorizing `identity:delete_project(project_id=09d3e6a988c14b7e860b96d58901271a)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:02.085715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: WARNING py.warnings [None req-e4f66306-d2f0-4a2f-a95e-90353044e231 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 30 09:46:02.085715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: warnings.warn(msg) Sep 30 09:46:02.085715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: Sep 30 09:46:02.086534 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f66306-d2f0-4a2f-a95e-90353044e231 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:02.092709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6572c255-b17f-4889-be4f-121de0a54f24 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:02.108649 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49abc633-ffc0-4e4d-9c7e-4e228d026fe4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:02.112239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49abc633-ffc0-4e4d-9c7e-4e228d026fe4 service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:02.112239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49abc633-ffc0-4e4d-9c7e-4e228d026fe4 service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:02.122636 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49abc633-ffc0-4e4d-9c7e-4e228d026fe4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:02.133908 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49abc633-ffc0-4e4d-9c7e-4e228d026fe4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:02.136519 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49abc633-ffc0-4e4d-9c7e-4e228d026fe4 service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:02.172770 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 846/1682] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:01 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 263 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:46:02.173756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49abc633-ffc0-4e4d-9c7e-4e228d026fe4 service nova] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:02.177359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-49abc633-ffc0-4e4d-9c7e-4e228d026fe4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:02.205783 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49abc633-ffc0-4e4d-9c7e-4e228d026fe4 service nova] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:02.215725 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a000904f-a146-46ce-8670-473506c62094 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:02.216391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a000904f-a146-46ce-8670-473506c62094 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:02.216695 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a000904f-a146-46ce-8670-473506c62094 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:02.216953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a000904f-a146-46ce-8670-473506c62094 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:02.237848 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a000904f-a146-46ce-8670-473506c62094 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:02.239874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a000904f-a146-46ce-8670-473506c62094 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:02.255110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 847/1683] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:02 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:46:02.423025 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e4f66306-d2f0-4a2f-a95e-90353044e231 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '09d3e6a988c14b7e860b96d58901271a', 'request_id': 'req-e4f66306-d2f0-4a2f-a95e-90353044e231'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:02.423025 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e4f66306-d2f0-4a2f-a95e-90353044e231 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '09d3e6a988c14b7e860b96d58901271a', 'request_id': 'req-e4f66306-d2f0-4a2f-a95e-90353044e231'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:02.423025 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e4f66306-d2f0-4a2f-a95e-90353044e231 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '09d3e6a988c14b7e860b96d58901271a', 'request_id': 'req-e4f66306-d2f0-4a2f-a95e-90353044e231'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:02.587036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:02.592596 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e4f66306-d2f0-4a2f-a95e-90353044e231 admin admin] The token cache is being invalidate because project 09d3e6a988c14b7e860b96d58901271a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:46:02.592596 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e4f66306-d2f0-4a2f-a95e-90353044e231 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:02.593755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e4f66306-d2f0-4a2f-a95e-90353044e231 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:02.595981 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 837/1684] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:46:01 2022] DELETE /identity/v3/projects/09d3e6a988c14b7e860b96d58901271a => generated 0 bytes in 600 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:02.600958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1e5857-8aa1-48a4-b05a-e785f53d5cb5 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:02.601802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1e5857-8aa1-48a4-b05a-e785f53d5cb5 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:02.602225 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1e5857-8aa1-48a4-b05a-e785f53d5cb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:02.666731 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-ba1e5857-8aa1-48a4-b05a-e785f53d5cb5 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:02.667406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ba1e5857-8aa1-48a4-b05a-e785f53d5cb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:02.724717 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:02.788408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f66306-d2f0-4a2f-a95e-90353044e231 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:02.791321 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e4f66306-d2f0-4a2f-a95e-90353044e231 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:02.843091 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 848/1685] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:02.851734 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389df85d-a651-49c2-af94-403724c31263 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:02.852688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389df85d-a651-49c2-af94-403724c31263 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:02.853041 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389df85d-a651-49c2-af94-403724c31263 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:02.890774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-389df85d-a651-49c2-af94-403724c31263 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:02.892984 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-389df85d-a651-49c2-af94-403724c31263 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:02.928005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 849/1686] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:02.942934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389df85d-a651-49c2-af94-403724c31263 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:02.979634 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389df85d-a651-49c2-af94-403724c31263 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:02.998756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b5b584-8ff7-4a5d-a9f5-924a8a1b6754 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:03.003222 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b5b584-8ff7-4a5d-a9f5-924a8a1b6754 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:03.005547 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b5b584-8ff7-4a5d-a9f5-924a8a1b6754 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:03.014820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b5b584-8ff7-4a5d-a9f5-924a8a1b6754 admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:03.014820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b5b584-8ff7-4a5d-a9f5-924a8a1b6754 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:46:03.014820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b5b584-8ff7-4a5d-a9f5-924a8a1b6754 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:03.020836 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b5b584-8ff7-4a5d-a9f5-924a8a1b6754 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:03.042386 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f66306-d2f0-4a2f-a95e-90353044e231 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:03.060016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ade81a3-fb95-4bb7-b126-1f029dbe9559 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:03.061529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ade81a3-fb95-4bb7-b126-1f029dbe9559 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:03.062027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ade81a3-fb95-4bb7-b126-1f029dbe9559 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:03.062522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ade81a3-fb95-4bb7-b126-1f029dbe9559 admin admin] PATH_INFO: `/v3/projects/15f01216eb594d88b37cb4377de3b831` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:03.065077 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ade81a3-fb95-4bb7-b126-1f029dbe9559 admin admin] RBAC: Authorizing `identity:delete_project(project_id=15f01216eb594d88b37cb4377de3b831)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:03.070295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ade81a3-fb95-4bb7-b126-1f029dbe9559 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:03.073425 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 850/1687] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:46:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:46:03.076406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b5b584-8ff7-4a5d-a9f5-924a8a1b6754 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:03.109671 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b5b584-8ff7-4a5d-a9f5-924a8a1b6754 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:03.131510 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f981f529-f325-4186-9b55-fb424914154b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:03.137781 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f981f529-f325-4186-9b55-fb424914154b admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:03.137781 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f981f529-f325-4186-9b55-fb424914154b admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:03.137781 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f981f529-f325-4186-9b55-fb424914154b admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:03.137781 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f981f529-f325-4186-9b55-fb424914154b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:03.146029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f981f529-f325-4186-9b55-fb424914154b admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:03.232008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-3ade81a3-fb95-4bb7-b126-1f029dbe9559 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '15f01216eb594d88b37cb4377de3b831', 'request_id': 'req-3ade81a3-fb95-4bb7-b126-1f029dbe9559'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:03.259505 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-3ade81a3-fb95-4bb7-b126-1f029dbe9559 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '15f01216eb594d88b37cb4377de3b831', 'request_id': 'req-3ade81a3-fb95-4bb7-b126-1f029dbe9559'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:03.287215 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-3ade81a3-fb95-4bb7-b126-1f029dbe9559 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '15f01216eb594d88b37cb4377de3b831', 'request_id': 'req-3ade81a3-fb95-4bb7-b126-1f029dbe9559'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:03.365091 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 851/1688] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:03 2022] POST /identity/v3/projects => generated 377 bytes in 286 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:03.366940 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f981f529-f325-4186-9b55-fb424914154b admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:03.396485 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-3ade81a3-fb95-4bb7-b126-1f029dbe9559 admin admin] The token cache is being invalidate because project 15f01216eb594d88b37cb4377de3b831 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:46:03.398309 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-3ade81a3-fb95-4bb7-b126-1f029dbe9559 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:03.400460 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-3ade81a3-fb95-4bb7-b126-1f029dbe9559 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:03.404321 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 838/1689] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:46:02 2022] DELETE /identity/v3/projects/15f01216eb594d88b37cb4377de3b831 => generated 0 bytes in 616 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:03.416529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ade81a3-fb95-4bb7-b126-1f029dbe9559 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:03.418362 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3ade81a3-fb95-4bb7-b126-1f029dbe9559 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:03.524576 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f981f529-f325-4186-9b55-fb424914154b admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:03.535885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14795076-c665-4c2a-8fc4-a597cc89118c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:03.536801 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14795076-c665-4c2a-8fc4-a597cc89118c admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:03.538404 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14795076-c665-4c2a-8fc4-a597cc89118c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:03.539532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14795076-c665-4c2a-8fc4-a597cc89118c admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:03.552674 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14795076-c665-4c2a-8fc4-a597cc89118c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:03.560856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14795076-c665-4c2a-8fc4-a597cc89118c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:03.581562 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ade81a3-fb95-4bb7-b126-1f029dbe9559 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:03.615396 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7840d1-a266-4aff-8a32-64129612af84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:03.617496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7840d1-a266-4aff-8a32-64129612af84 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:03.617496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7840d1-a266-4aff-8a32-64129612af84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:03.617496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7840d1-a266-4aff-8a32-64129612af84 admin admin] PATH_INFO: `/v3/users/200bc2f8aa4c4527a6c812c5836db2c5` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:03.618299 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7840d1-a266-4aff-8a32-64129612af84 admin admin] RBAC: Authorizing `identity:delete_user(user_id=200bc2f8aa4c4527a6c812c5836db2c5)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:03.622767 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7840d1-a266-4aff-8a32-64129612af84 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:03.721307 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 852/1690] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:46:03 2022] POST /identity/v3/users => generated 289 bytes in 355 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:46:03.725917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14795076-c665-4c2a-8fc4-a597cc89118c admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:03.733520 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-14795076-c665-4c2a-8fc4-a597cc89118c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:03.778716 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14795076-c665-4c2a-8fc4-a597cc89118c admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:03.795181 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949a989a-444c-4edb-b1e3-508b422aa98d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:03.795813 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949a989a-444c-4edb-b1e3-508b422aa98d admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:03.796126 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949a989a-444c-4edb-b1e3-508b422aa98d admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:03.797159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949a989a-444c-4edb-b1e3-508b422aa98d admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:03.797825 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949a989a-444c-4edb-b1e3-508b422aa98d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:03.799491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949a989a-444c-4edb-b1e3-508b422aa98d admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:03.822556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 853/1691] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:03 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:03.828456 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949a989a-444c-4edb-b1e3-508b422aa98d admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:03.862001 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949a989a-444c-4edb-b1e3-508b422aa98d admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:03.901127 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0b7840d1-a266-4aff-8a32-64129612af84 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '200bc2f8aa4c4527a6c812c5836db2c5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:03.901973 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e5477c-f140-42fb-b869-24940ee6c974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:03.903148 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e5477c-f140-42fb-b869-24940ee6c974 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:03.903612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0b7840d1-a266-4aff-8a32-64129612af84 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '200bc2f8aa4c4527a6c812c5836db2c5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:03.904337 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e5477c-f140-42fb-b869-24940ee6c974 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:03.904771 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0b7840d1-a266-4aff-8a32-64129612af84 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '200bc2f8aa4c4527a6c812c5836db2c5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:03.913678 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e5477c-f140-42fb-b869-24940ee6c974 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:03.914635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e5477c-f140-42fb-b869-24940ee6c974 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:03.919311 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e5477c-f140-42fb-b869-24940ee6c974 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:03.943921 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0b7840d1-a266-4aff-8a32-64129612af84 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '200bc2f8aa4c4527a6c812c5836db2c5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:03.968613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 854/1692] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:03 2022] GET /identity/v3/roles => generated 1420 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:03.972871 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e5477c-f140-42fb-b869-24940ee6c974 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:03.977930 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-15e5477c-f140-42fb-b869-24940ee6c974 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:04.016238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e5477c-f140-42fb-b869-24940ee6c974 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:04.024711 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2aa251-dad0-4912-a538-50553434ef57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:04.025792 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2aa251-dad0-4912-a538-50553434ef57 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:04.026898 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2aa251-dad0-4912-a538-50553434ef57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:04.027293 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2aa251-dad0-4912-a538-50553434ef57 admin admin] PATH_INFO: `/v3/projects/179cdac0ebe64e4ab4c01578c80c7845/users/9840276541c04f1abcf2e679da45ec33/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:04.038825 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0b7840d1-a266-4aff-8a32-64129612af84 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '200bc2f8aa4c4527a6c812c5836db2c5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:04.075368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2aa251-dad0-4912-a538-50553434ef57 admin admin] RBAC: Authorizing `identity:create_grant(project_id=179cdac0ebe64e4ab4c01578c80c7845, user_id=9840276541c04f1abcf2e679da45ec33, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:04.087166 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 839/1693] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:46:03 2022] DELETE /identity/v3/users/200bc2f8aa4c4527a6c812c5836db2c5 => generated 0 bytes in 675 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:04.093159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2aa251-dad0-4912-a538-50553434ef57 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:04.231520 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 855/1694] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:03 2022] PUT /identity/v3/projects/179cdac0ebe64e4ab4c01578c80c7845/users/9840276541c04f1abcf2e679da45ec33/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:04.242282 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdaea2c4-ef10-4651-9a6f-721d3e7a8f23 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:04.242682 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdaea2c4-ef10-4651-9a6f-721d3e7a8f23 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:04.243035 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdaea2c4-ef10-4651-9a6f-721d3e7a8f23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:04.314364 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-fdaea2c4-ef10-4651-9a6f-721d3e7a8f23 None None] MFA Rules not processed for user `9840276541c04f1abcf2e679da45ec33`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:04.333464 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fdaea2c4-ef10-4651-9a6f-721d3e7a8f23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:04.625780 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2aa251-dad0-4912-a538-50553434ef57 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:04.640663 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 840/1695] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:04 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 396 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:04.645396 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:04.679284 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:04.872039 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2aa251-dad0-4912-a538-50553434ef57 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:04.873433 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdaea2c4-ef10-4651-9a6f-721d3e7a8f23 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:04.877150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fdaea2c4-ef10-4651-9a6f-721d3e7a8f23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:04.881225 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9342a7f9-553e-486a-a7fc-7a5150ebf5d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:04.882222 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9342a7f9-553e-486a-a7fc-7a5150ebf5d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:04.882729 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9342a7f9-553e-486a-a7fc-7a5150ebf5d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:04.883165 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9342a7f9-553e-486a-a7fc-7a5150ebf5d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:04.883955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9342a7f9-553e-486a-a7fc-7a5150ebf5d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:04.887873 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9342a7f9-553e-486a-a7fc-7a5150ebf5d2 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:04.948244 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdaea2c4-ef10-4651-9a6f-721d3e7a8f23 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:04.962164 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf25607-abda-4493-abaf-e7dd068ad7de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:04.962928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf25607-abda-4493-abaf-e7dd068ad7de admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:04.963276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf25607-abda-4493-abaf-e7dd068ad7de admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:04.963765 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf25607-abda-4493-abaf-e7dd068ad7de admin admin] PATH_INFO: `/v3/projects/8ead447fbc024c66b835a0e1af4e746e` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:04.965001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf25607-abda-4493-abaf-e7dd068ad7de admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ead447fbc024c66b835a0e1af4e746e)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:04.967081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf25607-abda-4493-abaf-e7dd068ad7de admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:05.057019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 856/1696] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:04 2022] POST /identity/v3/projects => generated 377 bytes in 432 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:05.061011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9342a7f9-553e-486a-a7fc-7a5150ebf5d2 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:05.181814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9342a7f9-553e-486a-a7fc-7a5150ebf5d2 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:05.191717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cbf25607-abda-4493-abaf-e7dd068ad7de admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ead447fbc024c66b835a0e1af4e746e', 'request_id': 'req-cbf25607-abda-4493-abaf-e7dd068ad7de'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:05.249776 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cbf25607-abda-4493-abaf-e7dd068ad7de admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ead447fbc024c66b835a0e1af4e746e', 'request_id': 'req-cbf25607-abda-4493-abaf-e7dd068ad7de'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:05.304189 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cbf25607-abda-4493-abaf-e7dd068ad7de admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ead447fbc024c66b835a0e1af4e746e', 'request_id': 'req-cbf25607-abda-4493-abaf-e7dd068ad7de'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:05.404529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cbf25607-abda-4493-abaf-e7dd068ad7de admin admin] The token cache is being invalidate because project 8ead447fbc024c66b835a0e1af4e746e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:46:05.404825 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cbf25607-abda-4493-abaf-e7dd068ad7de admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:05.406523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cbf25607-abda-4493-abaf-e7dd068ad7de admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:05.410325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 841/1697] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:46:04 2022] DELETE /identity/v3/projects/8ead447fbc024c66b835a0e1af4e746e => generated 0 bytes in 539 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:05.418143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26883847-8954-476f-a8e7-4cebc92b35ed None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:05.426669 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363d6398-9db1-485e-976b-c57cbdeef427 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:05.427315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363d6398-9db1-485e-976b-c57cbdeef427 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:05.427603 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363d6398-9db1-485e-976b-c57cbdeef427 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:05.429268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363d6398-9db1-485e-976b-c57cbdeef427 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:05.438673 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26883847-8954-476f-a8e7-4cebc92b35ed None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:05.438963 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26883847-8954-476f-a8e7-4cebc92b35ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:05.446819 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363d6398-9db1-485e-976b-c57cbdeef427 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:05.451829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363d6398-9db1-485e-976b-c57cbdeef427 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:05.535663 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-26883847-8954-476f-a8e7-4cebc92b35ed None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:05.549054 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-26883847-8954-476f-a8e7-4cebc92b35ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:05.557795 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 842/1698] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:05.573838 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e80841-5d79-4c61-8bb9-bb5d167eea9d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:05.574134 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e80841-5d79-4c61-8bb9-bb5d167eea9d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:05.574618 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e80841-5d79-4c61-8bb9-bb5d167eea9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:05.615383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f5e80841-5d79-4c61-8bb9-bb5d167eea9d None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:05.617386 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f5e80841-5d79-4c61-8bb9-bb5d167eea9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:05.652315 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 843/1699] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:05.660304 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e80841-5d79-4c61-8bb9-bb5d167eea9d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:05.682848 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 857/1700] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:46:05 2022] POST /identity/v3/users => generated 295 bytes in 622 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:05.688648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363d6398-9db1-485e-976b-c57cbdeef427 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:05.691317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-363d6398-9db1-485e-976b-c57cbdeef427 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:05.730837 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363d6398-9db1-485e-976b-c57cbdeef427 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:05.743264 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e868a923-4aa5-40d7-b8cb-5ed65d9c8fae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:05.744180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e868a923-4aa5-40d7-b8cb-5ed65d9c8fae admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:05.746927 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e868a923-4aa5-40d7-b8cb-5ed65d9c8fae admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:05.748281 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e868a923-4aa5-40d7-b8cb-5ed65d9c8fae admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:05.749425 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e868a923-4aa5-40d7-b8cb-5ed65d9c8fae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:05.755845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e868a923-4aa5-40d7-b8cb-5ed65d9c8fae admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:05.762057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e80841-5d79-4c61-8bb9-bb5d167eea9d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:05.790529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbadd7d-ad9f-4a1e-a3ad-c9e432368841 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:05.794882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbadd7d-ad9f-4a1e-a3ad-c9e432368841 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:05.795171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbadd7d-ad9f-4a1e-a3ad-c9e432368841 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:05.807234 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbadd7d-ad9f-4a1e-a3ad-c9e432368841 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:05.809100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbadd7d-ad9f-4a1e-a3ad-c9e432368841 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:46:05.809422 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbadd7d-ad9f-4a1e-a3ad-c9e432368841 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:05.813050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 858/1701] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:05 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:05.820604 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbadd7d-ad9f-4a1e-a3ad-c9e432368841 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:05.820772 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e868a923-4aa5-40d7-b8cb-5ed65d9c8fae admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:05.865459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e868a923-4aa5-40d7-b8cb-5ed65d9c8fae admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:05.869175 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 844/1702] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:46:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:46:05.870716 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbadd7d-ad9f-4a1e-a3ad-c9e432368841 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:05.887727 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f836fb8-4ac3-468e-a063-ab2467c58f42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:05.888525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f836fb8-4ac3-468e-a063-ab2467c58f42 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:05.888900 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f836fb8-4ac3-468e-a063-ab2467c58f42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:05.889250 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f836fb8-4ac3-468e-a063-ab2467c58f42 admin admin] PATH_INFO: `/v3/projects/6c2541f2d9b84a7e8b07e26eab937f72/users/cd9ae696a0484c558fe101bc98682914/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:05.929340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbadd7d-ad9f-4a1e-a3ad-c9e432368841 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:05.948925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f836fb8-4ac3-468e-a063-ab2467c58f42 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6c2541f2d9b84a7e8b07e26eab937f72, user_id=cd9ae696a0484c558fe101bc98682914, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:05.951940 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f836fb8-4ac3-468e-a063-ab2467c58f42 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:05.955452 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:05.957139 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c658393-a0ba-4cc4-94d9-e5de994755b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:05.958556 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c658393-a0ba-4cc4-94d9-e5de994755b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:05.959026 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c658393-a0ba-4cc4-94d9-e5de994755b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:05.959749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c658393-a0ba-4cc4-94d9-e5de994755b3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:05.961852 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c658393-a0ba-4cc4-94d9-e5de994755b3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:05.970073 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c658393-a0ba-4cc4-94d9-e5de994755b3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:06.084677 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 859/1703] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:05 2022] PUT /identity/v3/projects/6c2541f2d9b84a7e8b07e26eab937f72/users/cd9ae696a0484c558fe101bc98682914/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:06.084677 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f836fb8-4ac3-468e-a063-ab2467c58f42 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:06.107794 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 845/1704] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:05 2022] POST /identity/v3/projects => generated 369 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:06.109943 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c658393-a0ba-4cc4-94d9-e5de994755b3 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:06.286284 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f836fb8-4ac3-468e-a063-ab2467c58f42 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:06.318844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c658393-a0ba-4cc4-94d9-e5de994755b3 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:06.346875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278dae66-c45a-4d52-83a8-c9b3e6d41042 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:06.347843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278dae66-c45a-4d52-83a8-c9b3e6d41042 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:06.348256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278dae66-c45a-4d52-83a8-c9b3e6d41042 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:06.348655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278dae66-c45a-4d52-83a8-c9b3e6d41042 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:06.349457 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278dae66-c45a-4d52-83a8-c9b3e6d41042 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:06.351582 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278dae66-c45a-4d52-83a8-c9b3e6d41042 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:06.421514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04382cb8-e52d-4a7e-9aa7-1667551a72f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:06.423923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04382cb8-e52d-4a7e-9aa7-1667551a72f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:06.424229 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04382cb8-e52d-4a7e-9aa7-1667551a72f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:06.424507 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04382cb8-e52d-4a7e-9aa7-1667551a72f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:06.425170 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04382cb8-e52d-4a7e-9aa7-1667551a72f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:06.429598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04382cb8-e52d-4a7e-9aa7-1667551a72f5 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:06.481875 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 860/1705] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:06 2022] GET /identity/v3/roles => generated 1420 bytes in 402 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:46:06.487096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04382cb8-e52d-4a7e-9aa7-1667551a72f5 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:06.506105 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 846/1706] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:46:06 2022] POST /identity/v3/users => generated 285 bytes in 395 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:06.506307 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04382cb8-e52d-4a7e-9aa7-1667551a72f5 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:06.507363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278dae66-c45a-4d52-83a8-c9b3e6d41042 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:06.523070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c098b1c2-0925-41bf-9e0b-c954875953e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:06.523943 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278dae66-c45a-4d52-83a8-c9b3e6d41042 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:06.524939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c098b1c2-0925-41bf-9e0b-c954875953e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:06.525218 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c098b1c2-0925-41bf-9e0b-c954875953e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:06.525824 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c098b1c2-0925-41bf-9e0b-c954875953e2 admin admin] PATH_INFO: `/v3/projects/6c2541f2d9b84a7e8b07e26eab937f72/users/cd9ae696a0484c558fe101bc98682914/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:06.528600 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c098b1c2-0925-41bf-9e0b-c954875953e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6c2541f2d9b84a7e8b07e26eab937f72, user_id=cd9ae696a0484c558fe101bc98682914, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:06.532952 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c098b1c2-0925-41bf-9e0b-c954875953e2 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:06.536440 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61e2e66-9fa7-498d-967f-a5435392b2a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:06.537084 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61e2e66-9fa7-498d-967f-a5435392b2a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:06.539575 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61e2e66-9fa7-498d-967f-a5435392b2a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:06.539872 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61e2e66-9fa7-498d-967f-a5435392b2a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:06.540628 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61e2e66-9fa7-498d-967f-a5435392b2a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:06.551628 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61e2e66-9fa7-498d-967f-a5435392b2a7 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:06.556276 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 861/1707] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:06 2022] PUT /identity/v3/projects/6c2541f2d9b84a7e8b07e26eab937f72/users/cd9ae696a0484c558fe101bc98682914/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:06.569705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c443c0c-7e04-4eff-8970-d345d8dc8ed3 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:06.569990 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c443c0c-7e04-4eff-8970-d345d8dc8ed3 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:06.570260 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c443c0c-7e04-4eff-8970-d345d8dc8ed3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:06.580766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 847/1708] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:06 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:06.582550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1583d427-6729-47f7-bc1a-e0cd1d6ba260 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:06.583006 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1583d427-6729-47f7-bc1a-e0cd1d6ba260 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:06.583747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1583d427-6729-47f7-bc1a-e0cd1d6ba260 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:06.610537 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-3c443c0c-7e04-4eff-8970-d345d8dc8ed3 None None] MFA Rules not processed for user `cd9ae696a0484c558fe101bc98682914`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:06.612937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-3c443c0c-7e04-4eff-8970-d345d8dc8ed3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:06.727722 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-1583d427-6729-47f7-bc1a-e0cd1d6ba260 None None] MFA Rules not processed for user `7d6e0545a40a436d8c63820b795551a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:06.762856 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1583d427-6729-47f7-bc1a-e0cd1d6ba260 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:06.847835 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:06.914719 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 862/1709] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:06 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 346 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:06.921134 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c443c0c-7e04-4eff-8970-d345d8dc8ed3 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:06.989665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 848/1710] 10.222.0.39 () {58 vars in 1009 bytes} [Fri Sep 30 09:46:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 408 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:06.990792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1583d427-6729-47f7-bc1a-e0cd1d6ba260 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:07.002373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1583d427-6729-47f7-bc1a-e0cd1d6ba260 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:07.238905 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c443c0c-7e04-4eff-8970-d345d8dc8ed3 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:07.242745 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1583d427-6729-47f7-bc1a-e0cd1d6ba260 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:07.253361 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71211664-f624-4cc6-b4ba-b6f0ffa88308 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:07.254004 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71211664-f624-4cc6-b4ba-b6f0ffa88308 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:07.254286 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71211664-f624-4cc6-b4ba-b6f0ffa88308 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:07.254561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71211664-f624-4cc6-b4ba-b6f0ffa88308 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:07.263380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3faa90-e466-441f-bd6d-a9f4d5612b78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:07.264023 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3faa90-e466-441f-bd6d-a9f4d5612b78 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:07.264306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3faa90-e466-441f-bd6d-a9f4d5612b78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:07.265107 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3faa90-e466-441f-bd6d-a9f4d5612b78 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:07.266571 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d3faa90-e466-441f-bd6d-a9f4d5612b78 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:07.270557 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d3faa90-e466-441f-bd6d-a9f4d5612b78 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:07.270834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-71211664-f624-4cc6-b4ba-b6f0ffa88308 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:07.298529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 863/1711] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:06 2022] GET /identity/v3/roles => generated 1420 bytes in 381 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:07.308061 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3faa90-e466-441f-bd6d-a9f4d5612b78 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:07.331162 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3faa90-e466-441f-bd6d-a9f4d5612b78 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:07.340939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3574154b-82cc-4cda-8cb6-0353f5fd0c37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:07.341632 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3574154b-82cc-4cda-8cb6-0353f5fd0c37 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:07.341913 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3574154b-82cc-4cda-8cb6-0353f5fd0c37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:07.342188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3574154b-82cc-4cda-8cb6-0353f5fd0c37 admin admin] PATH_INFO: `/v3/projects/e9ef243fe44845d3a4c631111a5c9ca8/users/2397cca4572f4539a6a0d9f10df2fca7/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:07.384013 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3574154b-82cc-4cda-8cb6-0353f5fd0c37 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9ef243fe44845d3a4c631111a5c9ca8, user_id=2397cca4572f4539a6a0d9f10df2fca7, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:07.387786 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3574154b-82cc-4cda-8cb6-0353f5fd0c37 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:07.456977 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:07.493176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 864/1712] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:07 2022] PUT /identity/v3/projects/e9ef243fe44845d3a4c631111a5c9ca8/users/2397cca4572f4539a6a0d9f10df2fca7/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:07.495148 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3574154b-82cc-4cda-8cb6-0353f5fd0c37 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:07.501099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-3574154b-82cc-4cda-8cb6-0353f5fd0c37 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:07.600499 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71211664-f624-4cc6-b4ba-b6f0ffa88308 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:07.607768 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71211664-f624-4cc6-b4ba-b6f0ffa88308 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:07.670869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 849/1713] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:06 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 680 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:46:07.676990 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2f5c20-dd0e-491c-8ecd-a199456d113c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:07.678091 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2f5c20-dd0e-491c-8ecd-a199456d113c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:07.679758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2f5c20-dd0e-491c-8ecd-a199456d113c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:07.769149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-0d2f5c20-dd0e-491c-8ecd-a199456d113c None None] MFA Rules not processed for user `8c4ced79c3ff4842b30b472acb555cd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:07.773233 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0d2f5c20-dd0e-491c-8ecd-a199456d113c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:07.893274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3574154b-82cc-4cda-8cb6-0353f5fd0c37 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:07.905618 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3f24cd-f829-4eb8-a39a-45b5ca7f9967 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e5ad921e6fa41c297696ec7279f0fc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:07.907185 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3f24cd-f829-4eb8-a39a-45b5ca7f9967 service swift] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:07.907701 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3f24cd-f829-4eb8-a39a-45b5ca7f9967 service swift] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:07.908191 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3f24cd-f829-4eb8-a39a-45b5ca7f9967 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:07.920378 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 850/1714] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:07 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 249 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:07.923336 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4aef16-6501-4aff-a130-c919cb13aa6f None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:07.923839 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4aef16-6501-4aff-a130-c919cb13aa6f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:07.924249 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4aef16-6501-4aff-a130-c919cb13aa6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:08.037527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7f4aef16-6501-4aff-a130-c919cb13aa6f None None] MFA Rules not processed for user `2397cca4572f4539a6a0d9f10df2fca7`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:08.043895 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7f4aef16-6501-4aff-a130-c919cb13aa6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:08.129556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3f24cd-f829-4eb8-a39a-45b5ca7f9967 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:08.129556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3f24cd-f829-4eb8-a39a-45b5ca7f9967 service swift] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:08.160412 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 865/1715] 10.222.0.39 () {60 vars in 1384 bytes} [Fri Sep 30 09:46:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 667 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 30 09:46:08.354376 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 851/1716] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:07 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 433 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:08.359167 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3f24cd-f829-4eb8-a39a-45b5ca7f9967 service swift] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:08.627845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3f24cd-f829-4eb8-a39a-45b5ca7f9967 service swift] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:08.653915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c64d53-2d6f-43fe-a2ea-c9fc5c5b2770 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:08.654551 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c64d53-2d6f-43fe-a2ea-c9fc5c5b2770 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:08.655086 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c64d53-2d6f-43fe-a2ea-c9fc5c5b2770 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:08.656003 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c64d53-2d6f-43fe-a2ea-c9fc5c5b2770 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:08.656762 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c64d53-2d6f-43fe-a2ea-c9fc5c5b2770 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:08.662862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c64d53-2d6f-43fe-a2ea-c9fc5c5b2770 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:08.827395 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 866/1717] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:08 2022] POST /identity/v3/projects => generated 367 bytes in 463 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:08.828375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4aef16-6501-4aff-a130-c919cb13aa6f None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:08.844581 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4aef16-6501-4aff-a130-c919cb13aa6f None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:08.926371 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:09.023599 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8939ad9f-1308-419b-b906-7606a13b8a0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:09.026487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8939ad9f-1308-419b-b906-7606a13b8a0f admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:09.026851 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8939ad9f-1308-419b-b906-7606a13b8a0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:09.027132 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8939ad9f-1308-419b-b906-7606a13b8a0f admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:09.032241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8939ad9f-1308-419b-b906-7606a13b8a0f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:09.034641 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8939ad9f-1308-419b-b906-7606a13b8a0f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:09.169344 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 852/1718] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:46:08 2022] POST /identity/v3/users => generated 290 bytes in 341 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:46:09.170014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c64d53-2d6f-43fe-a2ea-c9fc5c5b2770 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:09.191596 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c64d53-2d6f-43fe-a2ea-c9fc5c5b2770 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:09.192804 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:09.207835 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72032fdc-e8a5-4e4d-9119-af552462e98c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:09.209053 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72032fdc-e8a5-4e4d-9119-af552462e98c admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:09.209483 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72032fdc-e8a5-4e4d-9119-af552462e98c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:09.210005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72032fdc-e8a5-4e4d-9119-af552462e98c admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:09.211072 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72032fdc-e8a5-4e4d-9119-af552462e98c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:09.213044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72032fdc-e8a5-4e4d-9119-af552462e98c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:09.227254 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 867/1719] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:09 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:46:09.232354 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8939ad9f-1308-419b-b906-7606a13b8a0f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:09.254525 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8939ad9f-1308-419b-b906-7606a13b8a0f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:09.267947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1264de91-69ec-4986-b844-89214f31f544 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:09.267947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1264de91-69ec-4986-b844-89214f31f544 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:09.267947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1264de91-69ec-4986-b844-89214f31f544 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:09.267947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1264de91-69ec-4986-b844-89214f31f544 admin admin] PATH_INFO: `/v3/projects/1515f97c9abf462c8119127fa08851b2/users/500c08eae82140e5ab9f2224d43127b9/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:09.317182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1264de91-69ec-4986-b844-89214f31f544 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1515f97c9abf462c8119127fa08851b2, user_id=500c08eae82140e5ab9f2224d43127b9, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:09.320168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1264de91-69ec-4986-b844-89214f31f544 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:09.456711 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 853/1720] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:09 2022] PUT /identity/v3/projects/1515f97c9abf462c8119127fa08851b2/users/500c08eae82140e5ab9f2224d43127b9/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:09.459151 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72032fdc-e8a5-4e4d-9119-af552462e98c admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:09.476216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72032fdc-e8a5-4e4d-9119-af552462e98c admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:09.759927 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3241489-687c-4274-abf0-0dcf3f90d919 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:09.762079 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3241489-687c-4274-abf0-0dcf3f90d919 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:09.762079 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3241489-687c-4274-abf0-0dcf3f90d919 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:09.762079 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3241489-687c-4274-abf0-0dcf3f90d919 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:09.764204 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3241489-687c-4274-abf0-0dcf3f90d919 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:09.769947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3241489-687c-4274-abf0-0dcf3f90d919 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:09.789384 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acf761f-4d5c-4e88-9e5f-b3d01a984549 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:09.789795 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acf761f-4d5c-4e88-9e5f-b3d01a984549 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:09.790649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acf761f-4d5c-4e88-9e5f-b3d01a984549 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:09.798179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 868/1721] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:09 2022] GET /identity/v3/roles => generated 1420 bytes in 340 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:09.804672 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3241489-687c-4274-abf0-0dcf3f90d919 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:09.828213 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-3acf761f-4d5c-4e88-9e5f-b3d01a984549 None None] MFA Rules not processed for user `9266babd21414ae9998716e5a0478e9d`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:09.830922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3acf761f-4d5c-4e88-9e5f-b3d01a984549 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:09.850475 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3241489-687c-4274-abf0-0dcf3f90d919 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:09.894938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0414137-3e23-4076-a5c6-c648ead1ae6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:09.899179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0414137-3e23-4076-a5c6-c648ead1ae6c admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:09.906724 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0414137-3e23-4076-a5c6-c648ead1ae6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:09.907472 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0414137-3e23-4076-a5c6-c648ead1ae6c admin admin] PATH_INFO: `/v3/projects/1515f97c9abf462c8119127fa08851b2/users/500c08eae82140e5ab9f2224d43127b9/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:09.920200 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0414137-3e23-4076-a5c6-c648ead1ae6c admin admin] RBAC: Authorizing `identity:create_grant(project_id=1515f97c9abf462c8119127fa08851b2, user_id=500c08eae82140e5ab9f2224d43127b9, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:09.933515 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0414137-3e23-4076-a5c6-c648ead1ae6c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:09.989970 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 869/1722] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:09 2022] PUT /identity/v3/projects/1515f97c9abf462c8119127fa08851b2/users/500c08eae82140e5ab9f2224d43127b9/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:10.005093 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3515e53-7c6d-4946-9b11-7f9c0d3256dc None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:10.013012 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3515e53-7c6d-4946-9b11-7f9c0d3256dc None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:10.013012 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3515e53-7c6d-4946-9b11-7f9c0d3256dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:10.040145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 854/1723] 10.222.0.36 () {58 vars in 1015 bytes} [Fri Sep 30 09:46:09 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:10.106411 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-b3515e53-7c6d-4946-9b11-7f9c0d3256dc None None] MFA Rules not processed for user `500c08eae82140e5ab9f2224d43127b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:10.107967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b3515e53-7c6d-4946-9b11-7f9c0d3256dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:10.120536 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acf761f-4d5c-4e88-9e5f-b3d01a984549 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:10.123324 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3acf761f-4d5c-4e88-9e5f-b3d01a984549 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:10.157935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acf761f-4d5c-4e88-9e5f-b3d01a984549 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:10.306515 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:10.366260 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1f0d83-84f0-4767-ab76-e99702236216 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:10.370626 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1f0d83-84f0-4767-ab76-e99702236216 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:10.370773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 870/1724] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:10 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 366 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:10.372326 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1f0d83-84f0-4767-ab76-e99702236216 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:10.374436 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1f0d83-84f0-4767-ab76-e99702236216 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:10.384578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3515e53-7c6d-4946-9b11-7f9c0d3256dc None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:10.390157 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b3515e53-7c6d-4946-9b11-7f9c0d3256dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:10.429381 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1f0d83-84f0-4767-ab76-e99702236216 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:10.429381 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1f0d83-84f0-4767-ab76-e99702236216 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:10.463192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 855/1725] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:10 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:46:10.589880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3515e53-7c6d-4946-9b11-7f9c0d3256dc None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:10.602221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ec5645-202c-4097-8764-2085ab0ac53e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:10.603247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ec5645-202c-4097-8764-2085ab0ac53e service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:10.603358 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ec5645-202c-4097-8764-2085ab0ac53e service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:10.603774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ec5645-202c-4097-8764-2085ab0ac53e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:10.624279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ec5645-202c-4097-8764-2085ab0ac53e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:10.626068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ec5645-202c-4097-8764-2085ab0ac53e service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:10.639691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 871/1726] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:10 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:46:10.941214 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:10.991388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1f0d83-84f0-4767-ab76-e99702236216 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:11.008413 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1f0d83-84f0-4767-ab76-e99702236216 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:11.030724 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44683bb-81e7-4844-9fab-480f169f367c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:11.039123 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44683bb-81e7-4844-9fab-480f169f367c service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:11.039123 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44683bb-81e7-4844-9fab-480f169f367c service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:11.039123 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44683bb-81e7-4844-9fab-480f169f367c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:11.238486 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44683bb-81e7-4844-9fab-480f169f367c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:11.246924 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44683bb-81e7-4844-9fab-480f169f367c service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:11.274822 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 856/1727] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:10 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 284 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:46:11.467070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:12.662465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ec5645-202c-4097-8764-2085ab0ac53e service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:12.664077 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d4ec5645-202c-4097-8764-2085ab0ac53e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:12.758411 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ec5645-202c-4097-8764-2085ab0ac53e service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:12.763275 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99d5c78-1f62-4603-b789-d275ea2d9744 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:12.763986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99d5c78-1f62-4603-b789-d275ea2d9744 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:12.764369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99d5c78-1f62-4603-b789-d275ea2d9744 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:12.764714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99d5c78-1f62-4603-b789-d275ea2d9744 admin admin] PATH_INFO: `/v3/users/ab91a478674f4bc2804029c299138ce5` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:12.767451 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017bd4e0-80b1-4efd-8560-996deb99aaa2 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:12.767971 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017bd4e0-80b1-4efd-8560-996deb99aaa2 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:12.769366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017bd4e0-80b1-4efd-8560-996deb99aaa2 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:12.771057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 857/1728] 10.222.0.39 () {54 vars in 921 bytes} [Fri Sep 30 09:46:12 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 30 09:46:12.771772 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99d5c78-1f62-4603-b789-d275ea2d9744 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ab91a478674f4bc2804029c299138ce5)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:12.774663 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99d5c78-1f62-4603-b789-d275ea2d9744 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:12.780451 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f31419-3760-4ea9-a23c-685e5bbf2852 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:12.780451 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f31419-3760-4ea9-a23c-685e5bbf2852 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:12.780451 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f31419-3760-4ea9-a23c-685e5bbf2852 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:12.829149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-32f31419-3760-4ea9-a23c-685e5bbf2852 None None] MFA Rules not processed for user `7ff68fb87700423ab53c90520a6bd3b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:12.830410 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-32f31419-3760-4ea9-a23c-685e5bbf2852 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:12.899591 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 858/1729] 10.222.0.39 () {58 vars in 1018 bytes} [Fri Sep 30 09:46:12 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:12.913519 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f31419-3760-4ea9-a23c-685e5bbf2852 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:12.915987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-32f31419-3760-4ea9-a23c-685e5bbf2852 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:12.928861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f31419-3760-4ea9-a23c-685e5bbf2852 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:12.934269 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e1adf5-9033-4def-98d3-ddd152defc01 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:12.935502 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e1adf5-9033-4def-98d3-ddd152defc01 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:12.935787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e1adf5-9033-4def-98d3-ddd152defc01 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:12.936119 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e1adf5-9033-4def-98d3-ddd152defc01 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:12.944937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e1adf5-9033-4def-98d3-ddd152defc01 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:12.946526 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e1adf5-9033-4def-98d3-ddd152defc01 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:12.957353 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 859/1730] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:12 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:46:13.002963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a99d5c78-1f62-4603-b789-d275ea2d9744 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ab91a478674f4bc2804029c299138ce5'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:13.025898 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a99d5c78-1f62-4603-b789-d275ea2d9744 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ab91a478674f4bc2804029c299138ce5'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:13.027332 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a99d5c78-1f62-4603-b789-d275ea2d9744 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ab91a478674f4bc2804029c299138ce5'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:13.106508 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a99d5c78-1f62-4603-b789-d275ea2d9744 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ab91a478674f4bc2804029c299138ce5'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:13.114170 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a99d5c78-1f62-4603-b789-d275ea2d9744 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ab91a478674f4bc2804029c299138ce5'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:13.120927 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 872/1731] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:46:12 2022] DELETE /identity/v3/users/ab91a478674f4bc2804029c299138ce5 => generated 0 bytes in 459 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:13.125765 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501b0648-f8db-494c-a591-1f8a701683fe None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:13.126054 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501b0648-f8db-494c-a591-1f8a701683fe None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:13.126327 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501b0648-f8db-494c-a591-1f8a701683fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:13.145872 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-501b0648-f8db-494c-a591-1f8a701683fe None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:13.147110 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-501b0648-f8db-494c-a591-1f8a701683fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:13.193982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:13.221891 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 860/1732] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:13.232363 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99d5c78-1f62-4603-b789-d275ea2d9744 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:13.234037 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a99d5c78-1f62-4603-b789-d275ea2d9744 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:13.245206 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:13.293214 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99d5c78-1f62-4603-b789-d275ea2d9744 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:13.296846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a6573b-3807-46f0-ac0b-4393ccac435e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:13.297453 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a6573b-3807-46f0-ac0b-4393ccac435e service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:13.297777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a6573b-3807-46f0-ac0b-4393ccac435e service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:13.298046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a6573b-3807-46f0-ac0b-4393ccac435e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:13.307091 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a6573b-3807-46f0-ac0b-4393ccac435e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:13.308556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a6573b-3807-46f0-ac0b-4393ccac435e service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:13.318462 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 873/1733] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:46:13.862258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501b0648-f8db-494c-a591-1f8a701683fe None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:13.863833 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-501b0648-f8db-494c-a591-1f8a701683fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:13.879100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501b0648-f8db-494c-a591-1f8a701683fe None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:13.896385 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cfbafb4-df22-4739-8f43-39579529d1e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:13.897038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfbafb4-df22-4739-8f43-39579529d1e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:13.897330 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfbafb4-df22-4739-8f43-39579529d1e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:13.897600 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfbafb4-df22-4739-8f43-39579529d1e5 admin admin] PATH_INFO: `/v3/projects/c93c658159364808a249bdf8b2ccd752` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:13.899042 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cfbafb4-df22-4739-8f43-39579529d1e5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c93c658159364808a249bdf8b2ccd752)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:13.900940 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cfbafb4-df22-4739-8f43-39579529d1e5 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:14.046016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-3cfbafb4-df22-4739-8f43-39579529d1e5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c93c658159364808a249bdf8b2ccd752', 'request_id': 'req-3cfbafb4-df22-4739-8f43-39579529d1e5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:14.051836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-3cfbafb4-df22-4739-8f43-39579529d1e5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c93c658159364808a249bdf8b2ccd752', 'request_id': 'req-3cfbafb4-df22-4739-8f43-39579529d1e5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:14.058164 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-3cfbafb4-df22-4739-8f43-39579529d1e5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c93c658159364808a249bdf8b2ccd752', 'request_id': 'req-3cfbafb4-df22-4739-8f43-39579529d1e5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:14.128664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-3cfbafb4-df22-4739-8f43-39579529d1e5 admin admin] The token cache is being invalidate because project c93c658159364808a249bdf8b2ccd752 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:46:14.128953 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-3cfbafb4-df22-4739-8f43-39579529d1e5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:14.129565 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-3cfbafb4-df22-4739-8f43-39579529d1e5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:14.131070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 861/1734] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:46:13 2022] DELETE /identity/v3/projects/c93c658159364808a249bdf8b2ccd752 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:14.135055 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a6573b-3807-46f0-ac0b-4393ccac435e service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:14.145094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a6573b-3807-46f0-ac0b-4393ccac435e service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:14.211371 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fcc0e7-4855-47d0-acda-9dc182a825eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:14.212429 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fcc0e7-4855-47d0-acda-9dc182a825eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:14.213041 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fcc0e7-4855-47d0-acda-9dc182a825eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:14.213651 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fcc0e7-4855-47d0-acda-9dc182a825eb admin admin] PATH_INFO: `/v3/users/6c58c3b8a7aa46aea41571aff63590b3` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:14.215106 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fcc0e7-4855-47d0-acda-9dc182a825eb admin admin] RBAC: Authorizing `identity:delete_user(user_id=6c58c3b8a7aa46aea41571aff63590b3)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:14.218495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fcc0e7-4855-47d0-acda-9dc182a825eb admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:14.549585 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-62fcc0e7-4855-47d0-acda-9dc182a825eb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6c58c3b8a7aa46aea41571aff63590b3'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:14.567444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-62fcc0e7-4855-47d0-acda-9dc182a825eb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6c58c3b8a7aa46aea41571aff63590b3'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:14.568240 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-62fcc0e7-4855-47d0-acda-9dc182a825eb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6c58c3b8a7aa46aea41571aff63590b3'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:14.596492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cfbafb4-df22-4739-8f43-39579529d1e5 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:14.598159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3cfbafb4-df22-4739-8f43-39579529d1e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:14.653754 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-62fcc0e7-4855-47d0-acda-9dc182a825eb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6c58c3b8a7aa46aea41571aff63590b3'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:14.678233 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-62fcc0e7-4855-47d0-acda-9dc182a825eb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6c58c3b8a7aa46aea41571aff63590b3'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:14.690888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 874/1735] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:46:14 2022] DELETE /identity/v3/users/6c58c3b8a7aa46aea41571aff63590b3 => generated 0 bytes in 553 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:14.699192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cfbafb4-df22-4739-8f43-39579529d1e5 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:14.873873 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672513e3-6f90-4045-9390-67d37029e2c5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:14.874724 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672513e3-6f90-4045-9390-67d37029e2c5 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:14.874879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672513e3-6f90-4045-9390-67d37029e2c5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:14.875370 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672513e3-6f90-4045-9390-67d37029e2c5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:14.877498 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-672513e3-6f90-4045-9390-67d37029e2c5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:14.947268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:15.014963 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672513e3-6f90-4045-9390-67d37029e2c5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:15.016710 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672513e3-6f90-4045-9390-67d37029e2c5 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:15.053554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 862/1736] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:14 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 457 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:46:15.370659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fcc0e7-4855-47d0-acda-9dc182a825eb admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:15.373014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-62fcc0e7-4855-47d0-acda-9dc182a825eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:15.426414 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:15.484889 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fcc0e7-4855-47d0-acda-9dc182a825eb admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:15.490890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865c8130-e40b-449e-81ed-729973b7a174 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:15.491712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865c8130-e40b-449e-81ed-729973b7a174 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:15.492220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865c8130-e40b-449e-81ed-729973b7a174 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:15.492567 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865c8130-e40b-449e-81ed-729973b7a174 admin admin] PATH_INFO: `/v3/projects/dc3720bb351f43fc9ea163c42427bad6` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:15.493790 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-865c8130-e40b-449e-81ed-729973b7a174 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dc3720bb351f43fc9ea163c42427bad6)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:15.496172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-865c8130-e40b-449e-81ed-729973b7a174 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:15.660163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-865c8130-e40b-449e-81ed-729973b7a174 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dc3720bb351f43fc9ea163c42427bad6', 'request_id': 'req-865c8130-e40b-449e-81ed-729973b7a174'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:15.671846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-865c8130-e40b-449e-81ed-729973b7a174 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dc3720bb351f43fc9ea163c42427bad6', 'request_id': 'req-865c8130-e40b-449e-81ed-729973b7a174'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:15.682766 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-865c8130-e40b-449e-81ed-729973b7a174 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dc3720bb351f43fc9ea163c42427bad6', 'request_id': 'req-865c8130-e40b-449e-81ed-729973b7a174'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:15.786951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-865c8130-e40b-449e-81ed-729973b7a174 admin admin] The token cache is being invalidate because project dc3720bb351f43fc9ea163c42427bad6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:46:15.787328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-865c8130-e40b-449e-81ed-729973b7a174 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:15.792835 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-865c8130-e40b-449e-81ed-729973b7a174 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:15.792835 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 875/1737] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:46:15 2022] DELETE /identity/v3/projects/dc3720bb351f43fc9ea163c42427bad6 => generated 0 bytes in 420 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:15.795923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbf0b00-4daf-4bd4-bf6f-49556f765472 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:15.796232 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbf0b00-4daf-4bd4-bf6f-49556f765472 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:15.796644 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbf0b00-4daf-4bd4-bf6f-49556f765472 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:15.843428 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9cbf0b00-4daf-4bd4-bf6f-49556f765472 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:15.844794 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9cbf0b00-4daf-4bd4-bf6f-49556f765472 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:15.974018 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865c8130-e40b-449e-81ed-729973b7a174 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:15.975760 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-865c8130-e40b-449e-81ed-729973b7a174 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:16.007217 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 863/1738] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:16.016978 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a877f8-a8e8-4dc4-b2ef-4b3e3710d1eb None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:16.017344 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a877f8-a8e8-4dc4-b2ef-4b3e3710d1eb None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:16.017699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a877f8-a8e8-4dc4-b2ef-4b3e3710d1eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:16.042501 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-49a877f8-a8e8-4dc4-b2ef-4b3e3710d1eb None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:16.043898 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-49a877f8-a8e8-4dc4-b2ef-4b3e3710d1eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:16.055145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 864/1739] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:16.059684 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a877f8-a8e8-4dc4-b2ef-4b3e3710d1eb None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:16.086951 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a877f8-a8e8-4dc4-b2ef-4b3e3710d1eb None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:16.102512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f172d3-13a0-42ee-ba32-32333dd95792 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:16.104822 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f172d3-13a0-42ee-ba32-32333dd95792 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:16.105126 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f172d3-13a0-42ee-ba32-32333dd95792 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:16.105410 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f172d3-13a0-42ee-ba32-32333dd95792 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:16.106102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f172d3-13a0-42ee-ba32-32333dd95792 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:46:16.108630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f172d3-13a0-42ee-ba32-32333dd95792 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:16.108630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f172d3-13a0-42ee-ba32-32333dd95792 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:16.111135 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865c8130-e40b-449e-81ed-729973b7a174 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:16.131658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8a1e1f-23ed-40e8-8519-a00154812e77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:16.134472 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8a1e1f-23ed-40e8-8519-a00154812e77 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:16.138789 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8a1e1f-23ed-40e8-8519-a00154812e77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:16.139592 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 865/1740] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:46:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:46:16.139777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8a1e1f-23ed-40e8-8519-a00154812e77 admin admin] PATH_INFO: `/v3/users/1ce687dd153e4662a8bd34a0435d84e7` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:16.140850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8a1e1f-23ed-40e8-8519-a00154812e77 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1ce687dd153e4662a8bd34a0435d84e7)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:16.143482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8a1e1f-23ed-40e8-8519-a00154812e77 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:16.148210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f172d3-13a0-42ee-ba32-32333dd95792 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:16.162690 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f172d3-13a0-42ee-ba32-32333dd95792 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:16.168847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04052250-6c3b-4c62-a937-0eb3a2d4683b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:16.169558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04052250-6c3b-4c62-a937-0eb3a2d4683b admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:16.169914 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04052250-6c3b-4c62-a937-0eb3a2d4683b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:16.170528 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04052250-6c3b-4c62-a937-0eb3a2d4683b admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:16.171281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04052250-6c3b-4c62-a937-0eb3a2d4683b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:16.179167 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04052250-6c3b-4c62-a937-0eb3a2d4683b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:16.279265 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 866/1741] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:16 2022] POST /identity/v3/projects => generated 365 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:16.284078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04052250-6c3b-4c62-a937-0eb3a2d4683b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:16.409979 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04052250-6c3b-4c62-a937-0eb3a2d4683b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:16.434191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e94d07-d53a-4165-9bf1-54b8d36ef70b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:16.435345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e94d07-d53a-4165-9bf1-54b8d36ef70b admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:16.435345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e94d07-d53a-4165-9bf1-54b8d36ef70b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:16.447089 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e94d07-d53a-4165-9bf1-54b8d36ef70b admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:16.452226 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e94d07-d53a-4165-9bf1-54b8d36ef70b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:16.452352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-2f8a1e1f-23ed-40e8-8519-a00154812e77 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1ce687dd153e4662a8bd34a0435d84e7'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:16.452872 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e94d07-d53a-4165-9bf1-54b8d36ef70b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:16.495585 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-2f8a1e1f-23ed-40e8-8519-a00154812e77 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1ce687dd153e4662a8bd34a0435d84e7'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:16.567263 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-2f8a1e1f-23ed-40e8-8519-a00154812e77 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1ce687dd153e4662a8bd34a0435d84e7'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:17.272494 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-2f8a1e1f-23ed-40e8-8519-a00154812e77 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1ce687dd153e4662a8bd34a0435d84e7'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:17.277661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 867/1742] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:46:16 2022] POST /identity/v3/users => generated 283 bytes in 994 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:17.282237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e94d07-d53a-4165-9bf1-54b8d36ef70b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:17.283865 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b6e94d07-d53a-4165-9bf1-54b8d36ef70b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:17.288370 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-2f8a1e1f-23ed-40e8-8519-a00154812e77 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1ce687dd153e4662a8bd34a0435d84e7'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:17.298293 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 876/1743] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:46:15 2022] DELETE /identity/v3/users/1ce687dd153e4662a8bd34a0435d84e7 => generated 0 bytes in 1322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:17.304082 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e94d07-d53a-4165-9bf1-54b8d36ef70b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:17.313527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba2798c-7991-46e1-be05-e48cbe226ed7 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:17.313527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba2798c-7991-46e1-be05-e48cbe226ed7 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:17.313527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba2798c-7991-46e1-be05-e48cbe226ed7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:17.354580 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:17.355649 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-0ba2798c-7991-46e1-be05-e48cbe226ed7 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:17.356922 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0ba2798c-7991-46e1-be05-e48cbe226ed7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:17.495805 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2e3cbb-6cd9-4467-ad0c-029b7931ef4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:17.497451 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 877/1744] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:17.498312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2e3cbb-6cd9-4467-ad0c-029b7931ef4c admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:17.498312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2e3cbb-6cd9-4467-ad0c-029b7931ef4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:17.498312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2e3cbb-6cd9-4467-ad0c-029b7931ef4c admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:17.498312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2e3cbb-6cd9-4467-ad0c-029b7931ef4c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:17.501242 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2e3cbb-6cd9-4467-ad0c-029b7931ef4c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:17.540417 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba2798c-7991-46e1-be05-e48cbe226ed7 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:17.543150 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0ba2798c-7991-46e1-be05-e48cbe226ed7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:17.543329 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 868/1745] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:17 2022] GET /identity/v3/roles => generated 1420 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:17.550719 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2e3cbb-6cd9-4467-ad0c-029b7931ef4c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:17.590364 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2e3cbb-6cd9-4467-ad0c-029b7931ef4c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:17.625145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e207f9b-cb02-43cb-a110-9e18f538557e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:17.628093 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e207f9b-cb02-43cb-a110-9e18f538557e admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:17.628093 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e207f9b-cb02-43cb-a110-9e18f538557e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:17.628093 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e207f9b-cb02-43cb-a110-9e18f538557e admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:17.628093 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e207f9b-cb02-43cb-a110-9e18f538557e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:17.629309 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e207f9b-cb02-43cb-a110-9e18f538557e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:17.651383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 869/1746] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:17 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:17.652361 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e207f9b-cb02-43cb-a110-9e18f538557e admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:17.673152 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e207f9b-cb02-43cb-a110-9e18f538557e admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:17.692944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba2798c-7991-46e1-be05-e48cbe226ed7 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:17.699010 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0786b5-14ec-4f76-9f2d-78ea6cf74fdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:17.700083 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0786b5-14ec-4f76-9f2d-78ea6cf74fdc admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:17.701279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0786b5-14ec-4f76-9f2d-78ea6cf74fdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:17.702897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0786b5-14ec-4f76-9f2d-78ea6cf74fdc admin admin] PATH_INFO: `/v3/projects/9a686dae16a04996ad351ddb92875e0e/users/30170cffd964441a89299422c7738a8b/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:17.708401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a176fe36-5436-4fa3-ba7c-e836450d48f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:17.709461 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a176fe36-5436-4fa3-ba7c-e836450d48f8 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:17.709990 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a176fe36-5436-4fa3-ba7c-e836450d48f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:17.710998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a176fe36-5436-4fa3-ba7c-e836450d48f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:17.746456 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a176fe36-5436-4fa3-ba7c-e836450d48f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:17.746456 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a176fe36-5436-4fa3-ba7c-e836450d48f8 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:17.747045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0786b5-14ec-4f76-9f2d-78ea6cf74fdc admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a686dae16a04996ad351ddb92875e0e, user_id=30170cffd964441a89299422c7738a8b, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:17.747045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0786b5-14ec-4f76-9f2d-78ea6cf74fdc admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:17.747187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 878/1747] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:46:17.878930 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 870/1748] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:17 2022] PUT /identity/v3/projects/9a686dae16a04996ad351ddb92875e0e/users/30170cffd964441a89299422c7738a8b/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:17.898884 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa80b488-0ce6-473b-bc6e-1a5226cbc9d6 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:17.899172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa80b488-0ce6-473b-bc6e-1a5226cbc9d6 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:17.899450 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa80b488-0ce6-473b-bc6e-1a5226cbc9d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:17.938845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-aa80b488-0ce6-473b-bc6e-1a5226cbc9d6 None None] MFA Rules not processed for user `30170cffd964441a89299422c7738a8b`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:17.940155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-aa80b488-0ce6-473b-bc6e-1a5226cbc9d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:18.046062 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0786b5-14ec-4f76-9f2d-78ea6cf74fdc admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:18.046928 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 879/1749] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:17 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:18.129743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:18.145056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497023e2-beef-4b2c-a661-493ee99a7164 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:18.145342 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497023e2-beef-4b2c-a661-493ee99a7164 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:18.145617 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497023e2-beef-4b2c-a661-493ee99a7164 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:18.147241 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 880/1750] 10.222.0.39 () {54 vars in 921 bytes} [Fri Sep 30 09:46:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:46:18.147626 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0786b5-14ec-4f76-9f2d-78ea6cf74fdc admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:18.153761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25993399-7551-4256-8632-3163b35d299a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:18.154836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25993399-7551-4256-8632-3163b35d299a admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:18.154836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25993399-7551-4256-8632-3163b35d299a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:18.158289 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25993399-7551-4256-8632-3163b35d299a admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:18.158289 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25993399-7551-4256-8632-3163b35d299a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:18.158289 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25993399-7551-4256-8632-3163b35d299a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:18.168930 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867c6384-fb46-4ba6-8d8e-ebdeee9cba56 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:18.169903 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867c6384-fb46-4ba6-8d8e-ebdeee9cba56 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:18.170330 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867c6384-fb46-4ba6-8d8e-ebdeee9cba56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:18.193248 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-867c6384-fb46-4ba6-8d8e-ebdeee9cba56 None None] MFA Rules not processed for user `7ff68fb87700423ab53c90520a6bd3b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:18.194479 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-867c6384-fb46-4ba6-8d8e-ebdeee9cba56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:18.263853 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 871/1751] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:18 2022] POST /identity/v3/projects => generated 365 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:46:18.274861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25993399-7551-4256-8632-3163b35d299a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:18.283487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 881/1752] 10.222.0.39 () {58 vars in 1018 bytes} [Fri Sep 30 09:46:18 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:18.291194 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867c6384-fb46-4ba6-8d8e-ebdeee9cba56 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:18.292764 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-867c6384-fb46-4ba6-8d8e-ebdeee9cba56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:18.323179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867c6384-fb46-4ba6-8d8e-ebdeee9cba56 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:18.354257 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25993399-7551-4256-8632-3163b35d299a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:18.361222 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc70ad23-dbf2-455d-9849-4f67110c18f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:18.361630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc70ad23-dbf2-455d-9849-4f67110c18f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:18.361978 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc70ad23-dbf2-455d-9849-4f67110c18f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:18.362347 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc70ad23-dbf2-455d-9849-4f67110c18f1 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:18.363085 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc70ad23-dbf2-455d-9849-4f67110c18f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:18.365097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc70ad23-dbf2-455d-9849-4f67110c18f1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:18.405826 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fce3230-c118-48c5-a44a-f5edf314de43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:18.406481 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fce3230-c118-48c5-a44a-f5edf314de43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:18.406789 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fce3230-c118-48c5-a44a-f5edf314de43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:18.407069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fce3230-c118-48c5-a44a-f5edf314de43 admin admin] PATH_INFO: `/v3/users/2397cca4572f4539a6a0d9f10df2fca7` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:18.408401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fce3230-c118-48c5-a44a-f5edf314de43 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2397cca4572f4539a6a0d9f10df2fca7)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:18.410411 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fce3230-c118-48c5-a44a-f5edf314de43 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:18.443712 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 872/1753] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:46:18 2022] POST /identity/v3/users => generated 289 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:18.444534 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc70ad23-dbf2-455d-9849-4f67110c18f1 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:18.446050 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-cc70ad23-dbf2-455d-9849-4f67110c18f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:18.521895 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc70ad23-dbf2-455d-9849-4f67110c18f1 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:18.525290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575d4fce-febc-4b8a-b2e6-acbbd41754d7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:18.525915 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575d4fce-febc-4b8a-b2e6-acbbd41754d7 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:18.526202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575d4fce-febc-4b8a-b2e6-acbbd41754d7 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:18.526481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575d4fce-febc-4b8a-b2e6-acbbd41754d7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:18.535923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575d4fce-febc-4b8a-b2e6-acbbd41754d7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:18.537561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575d4fce-febc-4b8a-b2e6-acbbd41754d7 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:18.548904 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 873/1754] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:46:18.549735 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575d4fce-febc-4b8a-b2e6-acbbd41754d7 service nova] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:18.559865 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575d4fce-febc-4b8a-b2e6-acbbd41754d7 service nova] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:18.567622 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11f0629-a902-4648-b8ae-b1ef98fcb1ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:18.567873 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11f0629-a902-4648-b8ae-b1ef98fcb1ad admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:18.568191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11f0629-a902-4648-b8ae-b1ef98fcb1ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:18.568468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11f0629-a902-4648-b8ae-b1ef98fcb1ad admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:18.569101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11f0629-a902-4648-b8ae-b1ef98fcb1ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:18.570749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11f0629-a902-4648-b8ae-b1ef98fcb1ad admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:18.581396 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 874/1755] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:18 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:18.582346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11f0629-a902-4648-b8ae-b1ef98fcb1ad admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:18.584096 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b11f0629-a902-4648-b8ae-b1ef98fcb1ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:18.598437 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11f0629-a902-4648-b8ae-b1ef98fcb1ad admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:18.603485 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee60e625-0284-472a-9b5f-1d27b882b671 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:18.604130 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee60e625-0284-472a-9b5f-1d27b882b671 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:18.604411 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee60e625-0284-472a-9b5f-1d27b882b671 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:18.604686 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee60e625-0284-472a-9b5f-1d27b882b671 admin admin] PATH_INFO: `/v3/projects/a8f8c0108e1d45c6984325afff2c5fe9` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:18.605901 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee60e625-0284-472a-9b5f-1d27b882b671 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a8f8c0108e1d45c6984325afff2c5fe9)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:18.608436 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee60e625-0284-472a-9b5f-1d27b882b671 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:18.630415 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7fce3230-c118-48c5-a44a-f5edf314de43 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2397cca4572f4539a6a0d9f10df2fca7'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:18.643423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7fce3230-c118-48c5-a44a-f5edf314de43 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2397cca4572f4539a6a0d9f10df2fca7'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:18.644500 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7fce3230-c118-48c5-a44a-f5edf314de43 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2397cca4572f4539a6a0d9f10df2fca7'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:18.655110 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:18.688921 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ee60e625-0284-472a-9b5f-1d27b882b671 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a8f8c0108e1d45c6984325afff2c5fe9', 'request_id': 'req-ee60e625-0284-472a-9b5f-1d27b882b671'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:18.694374 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ee60e625-0284-472a-9b5f-1d27b882b671 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a8f8c0108e1d45c6984325afff2c5fe9', 'request_id': 'req-ee60e625-0284-472a-9b5f-1d27b882b671'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:18.694730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7fce3230-c118-48c5-a44a-f5edf314de43 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2397cca4572f4539a6a0d9f10df2fca7'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:18.703002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ee60e625-0284-472a-9b5f-1d27b882b671 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a8f8c0108e1d45c6984325afff2c5fe9', 'request_id': 'req-ee60e625-0284-472a-9b5f-1d27b882b671'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:18.703282 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7fce3230-c118-48c5-a44a-f5edf314de43 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2397cca4572f4539a6a0d9f10df2fca7'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:18.709323 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 882/1756] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:46:18 2022] DELETE /identity/v3/users/2397cca4572f4539a6a0d9f10df2fca7 => generated 0 bytes in 419 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:18.710085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fce3230-c118-48c5-a44a-f5edf314de43 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:18.711458 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7fce3230-c118-48c5-a44a-f5edf314de43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:18.776310 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fce3230-c118-48c5-a44a-f5edf314de43 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:18.782474 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7758c022-1461-4f2e-8635-ac9423c25eb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:18.782474 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7758c022-1461-4f2e-8635-ac9423c25eb1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:18.782474 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7758c022-1461-4f2e-8635-ac9423c25eb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:18.782474 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7758c022-1461-4f2e-8635-ac9423c25eb1 admin admin] PATH_INFO: `/v3/projects/7e9cca2169e84930adf5d41582bc75e0/users/efdb83070001469ea1bc5670094d9160/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:18.796132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7758c022-1461-4f2e-8635-ac9423c25eb1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e9cca2169e84930adf5d41582bc75e0, user_id=efdb83070001469ea1bc5670094d9160, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:18.798838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7758c022-1461-4f2e-8635-ac9423c25eb1 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:18.806957 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ee60e625-0284-472a-9b5f-1d27b882b671 admin admin] The token cache is being invalidate because project a8f8c0108e1d45c6984325afff2c5fe9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:46:18.808782 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ee60e625-0284-472a-9b5f-1d27b882b671 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:18.808782 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ee60e625-0284-472a-9b5f-1d27b882b671 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:18.809499 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 875/1757] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:46:18 2022] DELETE /identity/v3/projects/a8f8c0108e1d45c6984325afff2c5fe9 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:18.811351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee54efb-96d9-4775-9db9-f39b65f94d5a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:18.811636 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee54efb-96d9-4775-9db9-f39b65f94d5a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:18.811904 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee54efb-96d9-4775-9db9-f39b65f94d5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:18.830652 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-cee54efb-96d9-4775-9db9-f39b65f94d5a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:18.834156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-cee54efb-96d9-4775-9db9-f39b65f94d5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:18.839256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 876/1758] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:18.840552 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5d90e2-8880-40db-8e0f-945581f8691d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:18.840828 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5d90e2-8880-40db-8e0f-945581f8691d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:18.841095 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5d90e2-8880-40db-8e0f-945581f8691d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:18.859081 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 883/1759] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:18 2022] PUT /identity/v3/projects/7e9cca2169e84930adf5d41582bc75e0/users/efdb83070001469ea1bc5670094d9160/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:18.859907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-6e5d90e2-8880-40db-8e0f-945581f8691d None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:18.860029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7758c022-1461-4f2e-8635-ac9423c25eb1 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:18.861099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6e5d90e2-8880-40db-8e0f-945581f8691d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:18.861621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7758c022-1461-4f2e-8635-ac9423c25eb1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:18.922303 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7758c022-1461-4f2e-8635-ac9423c25eb1 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:18.923942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 877/1760] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:18.925016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5d90e2-8880-40db-8e0f-945581f8691d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:18.926738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6e5d90e2-8880-40db-8e0f-945581f8691d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:18.933243 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fecd26b-b9e6-4a74-9f0b-eedc0aa03d01 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:18.936937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fecd26b-b9e6-4a74-9f0b-eedc0aa03d01 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:18.936937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fecd26b-b9e6-4a74-9f0b-eedc0aa03d01 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:18.936937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fecd26b-b9e6-4a74-9f0b-eedc0aa03d01 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:18.940626 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5d90e2-8880-40db-8e0f-945581f8691d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:18.945468 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fecd26b-b9e6-4a74-9f0b-eedc0aa03d01 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:18.945621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c0318c-e29b-46fe-804d-12410875650c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:18.946095 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c0318c-e29b-46fe-804d-12410875650c admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:18.946375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c0318c-e29b-46fe-804d-12410875650c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:18.946664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c0318c-e29b-46fe-804d-12410875650c admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:18.947059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fecd26b-b9e6-4a74-9f0b-eedc0aa03d01 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:18.947303 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c0318c-e29b-46fe-804d-12410875650c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:18.949031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c0318c-e29b-46fe-804d-12410875650c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:18.960520 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 878/1761] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:18 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:18.965504 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 884/1762] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:46:18.965644 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e1ffc7-c6fa-4d50-bd79-54b1f8367ad5 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:18.965644 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e1ffc7-c6fa-4d50-bd79-54b1f8367ad5 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:18.965644 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e1ffc7-c6fa-4d50-bd79-54b1f8367ad5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:18.965962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fecd26b-b9e6-4a74-9f0b-eedc0aa03d01 service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:18.977533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fecd26b-b9e6-4a74-9f0b-eedc0aa03d01 service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:18.983702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2722081-38b2-4450-aefd-aca23cbbb292 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:18.984372 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2722081-38b2-4450-aefd-aca23cbbb292 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:18.984652 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2722081-38b2-4450-aefd-aca23cbbb292 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:18.984955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2722081-38b2-4450-aefd-aca23cbbb292 admin admin] PATH_INFO: `/v3/projects/7e9cca2169e84930adf5d41582bc75e0/users/efdb83070001469ea1bc5670094d9160/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:18.986979 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2722081-38b2-4450-aefd-aca23cbbb292 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e9cca2169e84930adf5d41582bc75e0, user_id=efdb83070001469ea1bc5670094d9160, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:18.988844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c1e1ffc7-c6fa-4d50-bd79-54b1f8367ad5 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:18.990221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2722081-38b2-4450-aefd-aca23cbbb292 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:18.990334 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c1e1ffc7-c6fa-4d50-bd79-54b1f8367ad5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:18.999225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 879/1763] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:19.003036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e1ffc7-c6fa-4d50-bd79-54b1f8367ad5 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:19.006255 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 885/1764] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:18 2022] PUT /identity/v3/projects/7e9cca2169e84930adf5d41582bc75e0/users/efdb83070001469ea1bc5670094d9160/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:19.019542 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e1ffc7-c6fa-4d50-bd79-54b1f8367ad5 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:19.021966 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89c2dba-8e99-4187-99a1-ef9e9280f4b8 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:19.022339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89c2dba-8e99-4187-99a1-ef9e9280f4b8 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:19.022668 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89c2dba-8e99-4187-99a1-ef9e9280f4b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:19.102114 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-a89c2dba-8e99-4187-99a1-ef9e9280f4b8 None None] MFA Rules not processed for user `efdb83070001469ea1bc5670094d9160`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:19.104041 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a89c2dba-8e99-4187-99a1-ef9e9280f4b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:19.147831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7129fce7-9798-4d2d-b487-f8d23f278217 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:19.148590 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7129fce7-9798-4d2d-b487-f8d23f278217 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:19.149193 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7129fce7-9798-4d2d-b487-f8d23f278217 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:19.149469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7129fce7-9798-4d2d-b487-f8d23f278217 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:19.150529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7129fce7-9798-4d2d-b487-f8d23f278217 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:46:19.150529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7129fce7-9798-4d2d-b487-f8d23f278217 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:19.152358 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7129fce7-9798-4d2d-b487-f8d23f278217 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:19.167192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 880/1765] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:46:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:46:19.180438 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7129fce7-9798-4d2d-b487-f8d23f278217 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:19.198196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7129fce7-9798-4d2d-b487-f8d23f278217 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:19.215502 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1043028e-d06e-439d-b311-b1b1191cea86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:19.219047 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1043028e-d06e-439d-b311-b1b1191cea86 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:19.219757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1043028e-d06e-439d-b311-b1b1191cea86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:19.220030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1043028e-d06e-439d-b311-b1b1191cea86 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:19.220685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1043028e-d06e-439d-b311-b1b1191cea86 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:19.227481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1043028e-d06e-439d-b311-b1b1191cea86 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:19.254088 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 886/1766] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:19 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:19.270580 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89c2dba-8e99-4187-99a1-ef9e9280f4b8 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:19.273129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a89c2dba-8e99-4187-99a1-ef9e9280f4b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:19.344901 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 881/1767] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:19 2022] POST /identity/v3/projects => generated 403 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:19.350905 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1043028e-d06e-439d-b311-b1b1191cea86 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:19.352561 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89c2dba-8e99-4187-99a1-ef9e9280f4b8 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:19.459756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1043028e-d06e-439d-b311-b1b1191cea86 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:19.460901 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270d045a-e4e1-43e9-9146-847b57d214cd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:19.461530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270d045a-e4e1-43e9-9146-847b57d214cd service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:19.461809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270d045a-e4e1-43e9-9146-847b57d214cd service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:19.462083 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270d045a-e4e1-43e9-9146-847b57d214cd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:19.468073 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cf63d4-cb2b-4bbf-b9c6-bd6a3060652a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:19.471352 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cf63d4-cb2b-4bbf-b9c6-bd6a3060652a admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:19.472011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cf63d4-cb2b-4bbf-b9c6-bd6a3060652a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:19.472584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cf63d4-cb2b-4bbf-b9c6-bd6a3060652a admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:19.474524 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cf63d4-cb2b-4bbf-b9c6-bd6a3060652a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:19.479490 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cf63d4-cb2b-4bbf-b9c6-bd6a3060652a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:19.561240 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270d045a-e4e1-43e9-9146-847b57d214cd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:19.565253 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270d045a-e4e1-43e9-9146-847b57d214cd service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:19.597267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 887/1768] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:19 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:46:19.598584 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270d045a-e4e1-43e9-9146-847b57d214cd service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:19.600650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-270d045a-e4e1-43e9-9146-847b57d214cd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:19.609707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 882/1769] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:46:19 2022] POST /identity/v3/users => generated 302 bytes in 256 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:19.611522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cf63d4-cb2b-4bbf-b9c6-bd6a3060652a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:19.633916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270d045a-e4e1-43e9-9146-847b57d214cd service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:19.653718 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793d941d-70c4-4902-b61b-5c3eda685a9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:19.653718 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793d941d-70c4-4902-b61b-5c3eda685a9d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:19.653718 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793d941d-70c4-4902-b61b-5c3eda685a9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:19.653718 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793d941d-70c4-4902-b61b-5c3eda685a9d admin admin] PATH_INFO: `/v3/projects/e9ef243fe44845d3a4c631111a5c9ca8` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:19.654306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793d941d-70c4-4902-b61b-5c3eda685a9d admin admin] RBAC: Authorizing `identity:delete_project(project_id=e9ef243fe44845d3a4c631111a5c9ca8)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:19.654306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793d941d-70c4-4902-b61b-5c3eda685a9d admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:19.656599 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cf63d4-cb2b-4bbf-b9c6-bd6a3060652a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:19.681091 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8407660a-4872-4b09-99fe-0a0bfdeb8375 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:19.681715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8407660a-4872-4b09-99fe-0a0bfdeb8375 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:19.682044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8407660a-4872-4b09-99fe-0a0bfdeb8375 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:19.683725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8407660a-4872-4b09-99fe-0a0bfdeb8375 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:19.683959 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8407660a-4872-4b09-99fe-0a0bfdeb8375 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:19.686253 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8407660a-4872-4b09-99fe-0a0bfdeb8375 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:19.702435 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 883/1770] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:19 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:19.707093 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8407660a-4872-4b09-99fe-0a0bfdeb8375 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:19.731223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8407660a-4872-4b09-99fe-0a0bfdeb8375 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:19.741004 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:DELETE=2,keystone:SELECT=97,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:19.745430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69b4231-4cc8-473a-82da-da4951496bbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:19.747326 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69b4231-4cc8-473a-82da-da4951496bbe admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:19.750467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69b4231-4cc8-473a-82da-da4951496bbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:19.750804 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69b4231-4cc8-473a-82da-da4951496bbe admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:19.751483 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69b4231-4cc8-473a-82da-da4951496bbe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:19.759976 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69b4231-4cc8-473a-82da-da4951496bbe admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:19.774162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:19.782848 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 884/1771] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:19 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:19.786813 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69b4231-4cc8-473a-82da-da4951496bbe admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:19.802118 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69b4231-4cc8-473a-82da-da4951496bbe admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:19.811748 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ca9c5a-141e-437a-9932-bf66641b98b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:19.812512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ca9c5a-141e-437a-9932-bf66641b98b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:19.813117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ca9c5a-141e-437a-9932-bf66641b98b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:19.813436 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ca9c5a-141e-437a-9932-bf66641b98b8 admin admin] PATH_INFO: `/v3/projects/067163b9757d46f7918f11725bdc007d/users/601990d6c9644b44bf34e0c3947a4280/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:19.840236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ca9c5a-141e-437a-9932-bf66641b98b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=067163b9757d46f7918f11725bdc007d, user_id=601990d6c9644b44bf34e0c3947a4280, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:19.843537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ca9c5a-141e-437a-9932-bf66641b98b8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:19.862848 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-793d941d-70c4-4902-b61b-5c3eda685a9d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e9ef243fe44845d3a4c631111a5c9ca8', 'request_id': 'req-793d941d-70c4-4902-b61b-5c3eda685a9d'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:19.874152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-793d941d-70c4-4902-b61b-5c3eda685a9d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e9ef243fe44845d3a4c631111a5c9ca8', 'request_id': 'req-793d941d-70c4-4902-b61b-5c3eda685a9d'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:19.884532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-793d941d-70c4-4902-b61b-5c3eda685a9d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e9ef243fe44845d3a4c631111a5c9ca8', 'request_id': 'req-793d941d-70c4-4902-b61b-5c3eda685a9d'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:19.917304 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 885/1772] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:19 2022] PUT /identity/v3/projects/067163b9757d46f7918f11725bdc007d/users/601990d6c9644b44bf34e0c3947a4280/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:19.918491 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ca9c5a-141e-437a-9932-bf66641b98b8 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:19.980582 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-793d941d-70c4-4902-b61b-5c3eda685a9d admin admin] The token cache is being invalidate because project e9ef243fe44845d3a4c631111a5c9ca8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:46:19.980958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-793d941d-70c4-4902-b61b-5c3eda685a9d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:19.981658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-793d941d-70c4-4902-b61b-5c3eda685a9d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:19.984143 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 888/1773] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:46:19 2022] DELETE /identity/v3/projects/e9ef243fe44845d3a4c631111a5c9ca8 => generated 0 bytes in 386 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:19.985458 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0faae91b-8a0d-4888-a6e1-c8a8eb6b7e14 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:19.985815 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0faae91b-8a0d-4888-a6e1-c8a8eb6b7e14 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:19.986077 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0faae91b-8a0d-4888-a6e1-c8a8eb6b7e14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:20.012574 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ca9c5a-141e-437a-9932-bf66641b98b8 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:20.026576 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-0faae91b-8a0d-4888-a6e1-c8a8eb6b7e14 None None] MFA Rules not processed for user `601990d6c9644b44bf34e0c3947a4280`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:20.027107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b57f8c-e55f-42f5-a23d-177cf73ac16b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:20.027829 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b57f8c-e55f-42f5-a23d-177cf73ac16b service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:20.027934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0faae91b-8a0d-4888-a6e1-c8a8eb6b7e14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:20.028174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b57f8c-e55f-42f5-a23d-177cf73ac16b service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:20.028494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b57f8c-e55f-42f5-a23d-177cf73ac16b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:20.031239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-27b57f8c-e55f-42f5-a23d-177cf73ac16b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:20.120053 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 889/1774] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:19 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:20.129612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b57f8c-e55f-42f5-a23d-177cf73ac16b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:20.132953 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b57f8c-e55f-42f5-a23d-177cf73ac16b service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:20.133136 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0faae91b-8a0d-4888-a6e1-c8a8eb6b7e14 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:20.136305 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0faae91b-8a0d-4888-a6e1-c8a8eb6b7e14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:20.151626 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 886/1775] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:19 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:46:20.152983 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b57f8c-e55f-42f5-a23d-177cf73ac16b service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:20.156226 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-27b57f8c-e55f-42f5-a23d-177cf73ac16b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:20.362510 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b57f8c-e55f-42f5-a23d-177cf73ac16b service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:20.367718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4940ca-14e2-4663-bbca-289bd8da167f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:20.368467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4940ca-14e2-4663-bbca-289bd8da167f admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:20.368846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4940ca-14e2-4663-bbca-289bd8da167f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:20.369186 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4940ca-14e2-4663-bbca-289bd8da167f admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:20.369927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4940ca-14e2-4663-bbca-289bd8da167f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:20.372392 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4940ca-14e2-4663-bbca-289bd8da167f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:20.477479 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0faae91b-8a0d-4888-a6e1-c8a8eb6b7e14 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:20.508230 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87150c71-870f-4dd7-883f-99d27ce41cb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:20.508898 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87150c71-870f-4dd7-883f-99d27ce41cb0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:20.509189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87150c71-870f-4dd7-883f-99d27ce41cb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:20.509467 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87150c71-870f-4dd7-883f-99d27ce41cb0 admin admin] PATH_INFO: `/v3/users/500c08eae82140e5ab9f2224d43127b9` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:20.514280 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 887/1776] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:20 2022] POST /identity/v3/projects => generated 405 bytes in 358 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:20.514812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87150c71-870f-4dd7-883f-99d27ce41cb0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=500c08eae82140e5ab9f2224d43127b9)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:20.514812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87150c71-870f-4dd7-883f-99d27ce41cb0 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:20.515292 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4940ca-14e2-4663-bbca-289bd8da167f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:20.810206 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4940ca-14e2-4663-bbca-289bd8da167f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:20.826163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-87150c71-870f-4dd7-883f-99d27ce41cb0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '500c08eae82140e5ab9f2224d43127b9'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:20.831903 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff35f53e-eb73-48a9-8628-764306fd2387 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:20.832688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff35f53e-eb73-48a9-8628-764306fd2387 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:20.832974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff35f53e-eb73-48a9-8628-764306fd2387 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:20.833252 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff35f53e-eb73-48a9-8628-764306fd2387 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:20.833901 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff35f53e-eb73-48a9-8628-764306fd2387 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:20.838519 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff35f53e-eb73-48a9-8628-764306fd2387 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:20.859329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-87150c71-870f-4dd7-883f-99d27ce41cb0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '500c08eae82140e5ab9f2224d43127b9'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:20.862439 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-87150c71-870f-4dd7-883f-99d27ce41cb0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '500c08eae82140e5ab9f2224d43127b9'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:20.947442 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:20.976162 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-87150c71-870f-4dd7-883f-99d27ce41cb0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '500c08eae82140e5ab9f2224d43127b9'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:20.992538 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 888/1777] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:46:20 2022] POST /identity/v3/users => generated 309 bytes in 478 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:20.996843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff35f53e-eb73-48a9-8628-764306fd2387 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:21.005035 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ff35f53e-eb73-48a9-8628-764306fd2387 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:21.010184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-87150c71-870f-4dd7-883f-99d27ce41cb0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '500c08eae82140e5ab9f2224d43127b9'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:21.035719 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 890/1778] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:46:20 2022] DELETE /identity/v3/users/500c08eae82140e5ab9f2224d43127b9 => generated 0 bytes in 915 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:21.040718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:21.054843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff35f53e-eb73-48a9-8628-764306fd2387 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:21.171133 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddf8660-9a0e-4675-b8f8-1470fd8538fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:21.173256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddf8660-9a0e-4675-b8f8-1470fd8538fc admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:21.173622 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddf8660-9a0e-4675-b8f8-1470fd8538fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:21.174158 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddf8660-9a0e-4675-b8f8-1470fd8538fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:21.175104 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddf8660-9a0e-4675-b8f8-1470fd8538fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:21.177560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddf8660-9a0e-4675-b8f8-1470fd8538fc admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:21.193677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 889/1779] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:20 2022] GET /identity/v3/roles => generated 1420 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:21.198951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87150c71-870f-4dd7-883f-99d27ce41cb0 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:21.221637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87150c71-870f-4dd7-883f-99d27ce41cb0 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:21.234176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a67899-ebcd-4aa3-9cb6-f4b2d9d72b8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:21.236508 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a67899-ebcd-4aa3-9cb6-f4b2d9d72b8e admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:21.236838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a67899-ebcd-4aa3-9cb6-f4b2d9d72b8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:21.237179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a67899-ebcd-4aa3-9cb6-f4b2d9d72b8e admin admin] PATH_INFO: `/v3/projects/df9900d26732437b9f775eb11d6352ce/users/937e02a381fa41dd8ad94d3acc403d65/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:21.275261 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a67899-ebcd-4aa3-9cb6-f4b2d9d72b8e admin admin] RBAC: Authorizing `identity:create_grant(project_id=df9900d26732437b9f775eb11d6352ce, user_id=937e02a381fa41dd8ad94d3acc403d65, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:21.281085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a67899-ebcd-4aa3-9cb6-f4b2d9d72b8e admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:21.373029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 891/1780] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:21 2022] PUT /identity/v3/projects/df9900d26732437b9f775eb11d6352ce/users/937e02a381fa41dd8ad94d3acc403d65/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:21.378645 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddf8660-9a0e-4675-b8f8-1470fd8538fc admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:21.394631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddf8660-9a0e-4675-b8f8-1470fd8538fc admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:21.573749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391c41d5-1377-45a8-86ea-f42913cdf839 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:21.578398 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391c41d5-1377-45a8-86ea-f42913cdf839 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:21.578964 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391c41d5-1377-45a8-86ea-f42913cdf839 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:21.579358 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391c41d5-1377-45a8-86ea-f42913cdf839 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:21.582802 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391c41d5-1377-45a8-86ea-f42913cdf839 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:21.587717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391c41d5-1377-45a8-86ea-f42913cdf839 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:21.610984 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 890/1781] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:21 2022] GET /identity/v3/roles => generated 1420 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:21.615408 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a67899-ebcd-4aa3-9cb6-f4b2d9d72b8e admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:21.635503 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a67899-ebcd-4aa3-9cb6-f4b2d9d72b8e admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:21.649030 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bc4206-493e-4a9f-95e7-fc045052e917 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:21.650111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bc4206-493e-4a9f-95e7-fc045052e917 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:21.650624 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bc4206-493e-4a9f-95e7-fc045052e917 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:21.651072 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bc4206-493e-4a9f-95e7-fc045052e917 admin admin] PATH_INFO: `/v3/projects/df9900d26732437b9f775eb11d6352ce/users/937e02a381fa41dd8ad94d3acc403d65/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:21.653874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bc4206-493e-4a9f-95e7-fc045052e917 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df9900d26732437b9f775eb11d6352ce, user_id=937e02a381fa41dd8ad94d3acc403d65, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:21.657404 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bc4206-493e-4a9f-95e7-fc045052e917 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:21.692265 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 892/1782] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:21 2022] PUT /identity/v3/projects/df9900d26732437b9f775eb11d6352ce/users/937e02a381fa41dd8ad94d3acc403d65/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:21.700891 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:21.703868 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2672c70a-c506-43a9-9103-af33b052d6e4 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:21.704333 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2672c70a-c506-43a9-9103-af33b052d6e4 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:21.704956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2672c70a-c506-43a9-9103-af33b052d6e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:21.780700 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-2672c70a-c506-43a9-9103-af33b052d6e4 None None] MFA Rules not processed for user `937e02a381fa41dd8ad94d3acc403d65`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:21.782384 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2672c70a-c506-43a9-9103-af33b052d6e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:21.946565 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:22.020370 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 891/1783] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:21 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 317 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:22.421265 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:22.572559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bc4206-493e-4a9f-95e7-fc045052e917 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:22.575816 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-53bc4206-493e-4a9f-95e7-fc045052e917 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:22.729334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bc4206-493e-4a9f-95e7-fc045052e917 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:22.737375 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ef01a5-4a53-4807-ae53-b6a6a09a8a54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:22.738042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ef01a5-4a53-4807-ae53-b6a6a09a8a54 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:22.738324 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ef01a5-4a53-4807-ae53-b6a6a09a8a54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:22.738624 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ef01a5-4a53-4807-ae53-b6a6a09a8a54 admin admin] PATH_INFO: `/v3/projects/1515f97c9abf462c8119127fa08851b2` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:22.740041 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ef01a5-4a53-4807-ae53-b6a6a09a8a54 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1515f97c9abf462c8119127fa08851b2)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:22.741998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ef01a5-4a53-4807-ae53-b6a6a09a8a54 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:22.768410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:22.890705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2672c70a-c506-43a9-9103-af33b052d6e4 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:22.892588 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2672c70a-c506-43a9-9103-af33b052d6e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:22.916334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-61ef01a5-4a53-4807-ae53-b6a6a09a8a54 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1515f97c9abf462c8119127fa08851b2', 'request_id': 'req-61ef01a5-4a53-4807-ae53-b6a6a09a8a54'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:22.936392 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-61ef01a5-4a53-4807-ae53-b6a6a09a8a54 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1515f97c9abf462c8119127fa08851b2', 'request_id': 'req-61ef01a5-4a53-4807-ae53-b6a6a09a8a54'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:22.945999 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-61ef01a5-4a53-4807-ae53-b6a6a09a8a54 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1515f97c9abf462c8119127fa08851b2', 'request_id': 'req-61ef01a5-4a53-4807-ae53-b6a6a09a8a54'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:23.028284 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2672c70a-c506-43a9-9103-af33b052d6e4 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:23.037044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-61ef01a5-4a53-4807-ae53-b6a6a09a8a54 admin admin] The token cache is being invalidate because project 1515f97c9abf462c8119127fa08851b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:46:23.037044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-61ef01a5-4a53-4807-ae53-b6a6a09a8a54 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:23.037044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-61ef01a5-4a53-4807-ae53-b6a6a09a8a54 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:23.040370 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 893/1784] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:46:22 2022] DELETE /identity/v3/projects/1515f97c9abf462c8119127fa08851b2 => generated 0 bytes in 468 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:23.045368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd879328-d798-4918-99cc-1f6c112dc2c1 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:23.055302 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd879328-d798-4918-99cc-1f6c112dc2c1 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:23.055718 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd879328-d798-4918-99cc-1f6c112dc2c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:23.132955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-dd879328-d798-4918-99cc-1f6c112dc2c1 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:23.132955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-dd879328-d798-4918-99cc-1f6c112dc2c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:23.203539 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dea91f0-ab76-4f44-af90-ea5de1b8eaf0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:23.204395 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dea91f0-ab76-4f44-af90-ea5de1b8eaf0 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:23.204869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dea91f0-ab76-4f44-af90-ea5de1b8eaf0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:23.205344 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dea91f0-ab76-4f44-af90-ea5de1b8eaf0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:23.218061 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4dea91f0-ab76-4f44-af90-ea5de1b8eaf0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:23.271914 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 894/1785] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:23.299315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbacbab-4033-4076-91a3-0504debc4373 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:23.299315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbacbab-4033-4076-91a3-0504debc4373 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:23.299315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbacbab-4033-4076-91a3-0504debc4373 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:23.343829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-8dbacbab-4033-4076-91a3-0504debc4373 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:23.346850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-8dbacbab-4033-4076-91a3-0504debc4373 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:23.360551 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dea91f0-ab76-4f44-af90-ea5de1b8eaf0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:23.362880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dea91f0-ab76-4f44-af90-ea5de1b8eaf0 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:23.371909 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 895/1786] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:23.372990 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbacbab-4033-4076-91a3-0504debc4373 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:23.394328 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 892/1787] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:22 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 504 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:46:23.418402 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbacbab-4033-4076-91a3-0504debc4373 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:23.424846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c18b21-79a0-44a2-9245-d087994783f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:23.426070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c18b21-79a0-44a2-9245-d087994783f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:23.426527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c18b21-79a0-44a2-9245-d087994783f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:23.427375 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c18b21-79a0-44a2-9245-d087994783f1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:23.428130 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c18b21-79a0-44a2-9245-d087994783f1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:46:23.428507 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c18b21-79a0-44a2-9245-d087994783f1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:23.430309 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c18b21-79a0-44a2-9245-d087994783f1 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:23.474711 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 896/1788] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:46:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:46:23.480146 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dea91f0-ab76-4f44-af90-ea5de1b8eaf0 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:23.500762 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dea91f0-ab76-4f44-af90-ea5de1b8eaf0 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:23.518039 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fab0592-d9cf-44e2-aa9b-aa485b949b23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:23.518681 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fab0592-d9cf-44e2-aa9b-aa485b949b23 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:23.518893 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fab0592-d9cf-44e2-aa9b-aa485b949b23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:23.519170 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fab0592-d9cf-44e2-aa9b-aa485b949b23 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:23.519811 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fab0592-d9cf-44e2-aa9b-aa485b949b23 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:23.525098 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fab0592-d9cf-44e2-aa9b-aa485b949b23 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:23.704861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 893/1789] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:23 2022] POST /identity/v3/projects => generated 387 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:46:23.706217 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c18b21-79a0-44a2-9245-d087994783f1 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:23.735164 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c18b21-79a0-44a2-9245-d087994783f1 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:23.746739 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fab0592-d9cf-44e2-aa9b-aa485b949b23 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:23.751713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8fab0592-d9cf-44e2-aa9b-aa485b949b23 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:23.836812 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:23.939929 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e110090-e87e-4eb3-b022-4d48f62d35ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:23.941743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e110090-e87e-4eb3-b022-4d48f62d35ce admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:23.943331 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e110090-e87e-4eb3-b022-4d48f62d35ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:23.943617 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e110090-e87e-4eb3-b022-4d48f62d35ce admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:23.944455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e110090-e87e-4eb3-b022-4d48f62d35ce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:23.947317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e110090-e87e-4eb3-b022-4d48f62d35ce admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:23.954849 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fab0592-d9cf-44e2-aa9b-aa485b949b23 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:23.966824 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5256e93d-9775-4db6-b579-8ad5ec65698b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:23.966824 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5256e93d-9775-4db6-b579-8ad5ec65698b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:23.966824 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5256e93d-9775-4db6-b579-8ad5ec65698b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:23.966824 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5256e93d-9775-4db6-b579-8ad5ec65698b admin admin] PATH_INFO: `/v3/users/601990d6c9644b44bf34e0c3947a4280` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:23.967533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5256e93d-9775-4db6-b579-8ad5ec65698b admin admin] RBAC: Authorizing `identity:delete_user(user_id=601990d6c9644b44bf34e0c3947a4280)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:23.967533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5256e93d-9775-4db6-b579-8ad5ec65698b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:24.073533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 897/1790] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:46:23 2022] POST /identity/v3/users => generated 294 bytes in 368 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:24.075683 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e110090-e87e-4eb3-b022-4d48f62d35ce admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:24.086263 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e110090-e87e-4eb3-b022-4d48f62d35ce admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:24.091283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83722c7c-a71c-41ad-99f2-ebb5254b9b3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:24.092616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83722c7c-a71c-41ad-99f2-ebb5254b9b3d admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:24.092900 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83722c7c-a71c-41ad-99f2-ebb5254b9b3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:24.093175 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83722c7c-a71c-41ad-99f2-ebb5254b9b3d admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:24.093818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83722c7c-a71c-41ad-99f2-ebb5254b9b3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:24.095450 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83722c7c-a71c-41ad-99f2-ebb5254b9b3d admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:24.106105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 898/1791] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:24 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:24.110723 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83722c7c-a71c-41ad-99f2-ebb5254b9b3d admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:24.120441 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83722c7c-a71c-41ad-99f2-ebb5254b9b3d admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:24.125710 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9bf57e0-38a9-4ab2-8c2d-f1d317a3cd13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:24.126354 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bf57e0-38a9-4ab2-8c2d-f1d317a3cd13 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:24.126663 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bf57e0-38a9-4ab2-8c2d-f1d317a3cd13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:24.126938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bf57e0-38a9-4ab2-8c2d-f1d317a3cd13 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:24.127570 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9bf57e0-38a9-4ab2-8c2d-f1d317a3cd13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:24.129145 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9bf57e0-38a9-4ab2-8c2d-f1d317a3cd13 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:24.139060 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 899/1792] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:24 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:46:24.142923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9bf57e0-38a9-4ab2-8c2d-f1d317a3cd13 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:24.152183 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9bf57e0-38a9-4ab2-8c2d-f1d317a3cd13 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:24.157429 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ace0f0f-d598-4bb5-9ff7-652f82b11735 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:24.158056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ace0f0f-d598-4bb5-9ff7-652f82b11735 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:24.158334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ace0f0f-d598-4bb5-9ff7-652f82b11735 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:24.158680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ace0f0f-d598-4bb5-9ff7-652f82b11735 admin admin] PATH_INFO: `/v3/projects/2cb74e9c881a43189117a06e1294e083/users/3477b16bb6ca47609fb29e21f52e08ab/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:24.173591 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ace0f0f-d598-4bb5-9ff7-652f82b11735 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2cb74e9c881a43189117a06e1294e083, user_id=3477b16bb6ca47609fb29e21f52e08ab, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:24.176288 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ace0f0f-d598-4bb5-9ff7-652f82b11735 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:24.274670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 900/1793] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:24 2022] PUT /identity/v3/projects/2cb74e9c881a43189117a06e1294e083/users/3477b16bb6ca47609fb29e21f52e08ab/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:24.286914 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087615d8-e45a-44da-9a8d-b53437b281e4 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:24.287194 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087615d8-e45a-44da-9a8d-b53437b281e4 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:24.287468 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087615d8-e45a-44da-9a8d-b53437b281e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:24.323699 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-087615d8-e45a-44da-9a8d-b53437b281e4 None None] MFA Rules not processed for user `3477b16bb6ca47609fb29e21f52e08ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:24.325064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-087615d8-e45a-44da-9a8d-b53437b281e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:24.335092 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5256e93d-9775-4db6-b579-8ad5ec65698b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '601990d6c9644b44bf34e0c3947a4280'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:24.351140 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5256e93d-9775-4db6-b579-8ad5ec65698b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '601990d6c9644b44bf34e0c3947a4280'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:24.352960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5256e93d-9775-4db6-b579-8ad5ec65698b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '601990d6c9644b44bf34e0c3947a4280'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:24.363910 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5256e93d-9775-4db6-b579-8ad5ec65698b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '601990d6c9644b44bf34e0c3947a4280'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:24.404566 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 901/1794] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:24 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:24.410034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087615d8-e45a-44da-9a8d-b53437b281e4 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:24.412131 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-087615d8-e45a-44da-9a8d-b53437b281e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:24.488551 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5256e93d-9775-4db6-b579-8ad5ec65698b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '601990d6c9644b44bf34e0c3947a4280'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:24.516920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087615d8-e45a-44da-9a8d-b53437b281e4 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:24.520149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 894/1795] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:46:23 2022] DELETE /identity/v3/users/601990d6c9644b44bf34e0c3947a4280 => generated 0 bytes in 774 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:24.523982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9e6a24-daad-4f81-9680-cda4cec431bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:24.525099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9e6a24-daad-4f81-9680-cda4cec431bc admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:24.525528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9e6a24-daad-4f81-9680-cda4cec431bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:24.525959 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9e6a24-daad-4f81-9680-cda4cec431bc admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:24.526454 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99258ec6-0d60-45a2-a1fe-8c6242826a49 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:24.527012 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9e6a24-daad-4f81-9680-cda4cec431bc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:24.527867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99258ec6-0d60-45a2-a1fe-8c6242826a49 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:24.528198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99258ec6-0d60-45a2-a1fe-8c6242826a49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:24.530468 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9e6a24-daad-4f81-9680-cda4cec431bc admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:24.563718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-99258ec6-0d60-45a2-a1fe-8c6242826a49 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:24.565679 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-99258ec6-0d60-45a2-a1fe-8c6242826a49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:24.653195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 895/1796] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:24.662958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99258ec6-0d60-45a2-a1fe-8c6242826a49 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:24.664423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-99258ec6-0d60-45a2-a1fe-8c6242826a49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:24.677527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:24.721941 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99258ec6-0d60-45a2-a1fe-8c6242826a49 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:24.728793 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 902/1797] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:24 2022] POST /identity/v3/projects => generated 387 bytes in 316 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:24.728958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc892e30-0dd5-470f-9e41-88d4e00bfb89 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:24.728958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc892e30-0dd5-470f-9e41-88d4e00bfb89 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:24.728958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc892e30-0dd5-470f-9e41-88d4e00bfb89 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:24.728958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc892e30-0dd5-470f-9e41-88d4e00bfb89 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:24.730391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9e6a24-daad-4f81-9680-cda4cec431bc admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:24.738285 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc892e30-0dd5-470f-9e41-88d4e00bfb89 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:24.740412 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc892e30-0dd5-470f-9e41-88d4e00bfb89 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:24.754229 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 896/1798] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:46:24.797967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9e6a24-daad-4f81-9680-cda4cec431bc admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:24.803566 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ea49df-3a58-4bd6-be99-87fc42256059 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:24.806288 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ea49df-3a58-4bd6-be99-87fc42256059 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:24.806288 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ea49df-3a58-4bd6-be99-87fc42256059 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:24.806288 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ea49df-3a58-4bd6-be99-87fc42256059 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:24.806288 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ea49df-3a58-4bd6-be99-87fc42256059 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:24.807928 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ea49df-3a58-4bd6-be99-87fc42256059 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:24.913122 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 903/1799] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:46:24 2022] POST /identity/v3/users => generated 300 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:46:24.917090 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc892e30-0dd5-470f-9e41-88d4e00bfb89 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:24.928346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc892e30-0dd5-470f-9e41-88d4e00bfb89 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:24.960965 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0dfb6d-11a9-470d-84c8-76ce91cdb4f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:24.962651 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0dfb6d-11a9-470d-84c8-76ce91cdb4f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:24.962651 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0dfb6d-11a9-470d-84c8-76ce91cdb4f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:24.962918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0dfb6d-11a9-470d-84c8-76ce91cdb4f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:24.963755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0dfb6d-11a9-470d-84c8-76ce91cdb4f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:24.966238 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0dfb6d-11a9-470d-84c8-76ce91cdb4f9 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:24.990262 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 897/1800] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:24 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:24.999867 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ea49df-3a58-4bd6-be99-87fc42256059 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:25.016731 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ea49df-3a58-4bd6-be99-87fc42256059 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:25.024115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1efce2a-f87a-43de-b91b-9040b887d7fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:25.024828 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1efce2a-f87a-43de-b91b-9040b887d7fd admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:25.025104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1efce2a-f87a-43de-b91b-9040b887d7fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:25.025378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1efce2a-f87a-43de-b91b-9040b887d7fd admin admin] PATH_INFO: `/v3/projects/0d17d9f4c2d442f08f77adb7703aa309/users/4256952c2eb74fdca049486509def40c/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:25.055819 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1efce2a-f87a-43de-b91b-9040b887d7fd admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d17d9f4c2d442f08f77adb7703aa309, user_id=4256952c2eb74fdca049486509def40c, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:25.059361 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1efce2a-f87a-43de-b91b-9040b887d7fd admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:25.156577 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 904/1801] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:24 2022] PUT /identity/v3/projects/0d17d9f4c2d442f08f77adb7703aa309/users/4256952c2eb74fdca049486509def40c/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:25.157330 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0dfb6d-11a9-470d-84c8-76ce91cdb4f9 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:25.200495 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0dfb6d-11a9-470d-84c8-76ce91cdb4f9 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:25.273236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b98e34-ae0e-4e84-b754-9a7c86290a99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:25.274745 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b98e34-ae0e-4e84-b754-9a7c86290a99 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:25.274745 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b98e34-ae0e-4e84-b754-9a7c86290a99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:25.274745 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b98e34-ae0e-4e84-b754-9a7c86290a99 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:25.275110 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b98e34-ae0e-4e84-b754-9a7c86290a99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:25.276716 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b98e34-ae0e-4e84-b754-9a7c86290a99 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:25.288243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 898/1802] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:25 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:46:25.292511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1efce2a-f87a-43de-b91b-9040b887d7fd admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:25.306412 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1efce2a-f87a-43de-b91b-9040b887d7fd admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:25.311716 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0d075b-6e69-4002-bd25-a13cc815bfd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:25.312486 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0d075b-6e69-4002-bd25-a13cc815bfd1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:25.312771 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0d075b-6e69-4002-bd25-a13cc815bfd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:25.313557 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0d075b-6e69-4002-bd25-a13cc815bfd1 admin admin] PATH_INFO: `/v3/projects/0d17d9f4c2d442f08f77adb7703aa309/users/4256952c2eb74fdca049486509def40c/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:25.314992 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0d075b-6e69-4002-bd25-a13cc815bfd1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d17d9f4c2d442f08f77adb7703aa309, user_id=4256952c2eb74fdca049486509def40c, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:25.317746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0d075b-6e69-4002-bd25-a13cc815bfd1 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:25.339495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 905/1803] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:25 2022] PUT /identity/v3/projects/0d17d9f4c2d442f08f77adb7703aa309/users/4256952c2eb74fdca049486509def40c/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:25.350984 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883abc37-9fdd-4ea6-adfb-de0434fbfb9d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:25.351265 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883abc37-9fdd-4ea6-adfb-de0434fbfb9d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:25.351544 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883abc37-9fdd-4ea6-adfb-de0434fbfb9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:25.362775 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:25.374099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0d075b-6e69-4002-bd25-a13cc815bfd1 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:25.376869 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-4d0d075b-6e69-4002-bd25-a13cc815bfd1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:25.401178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-883abc37-9fdd-4ea6-adfb-de0434fbfb9d None None] MFA Rules not processed for user `4256952c2eb74fdca049486509def40c`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:25.402876 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-883abc37-9fdd-4ea6-adfb-de0434fbfb9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:25.468192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0d075b-6e69-4002-bd25-a13cc815bfd1 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:25.474540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08826e27-8d87-4b75-851c-be6d026726f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:25.475401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08826e27-8d87-4b75-851c-be6d026726f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:25.475861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08826e27-8d87-4b75-851c-be6d026726f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:25.476495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08826e27-8d87-4b75-851c-be6d026726f8 admin admin] PATH_INFO: `/v3/projects/067163b9757d46f7918f11725bdc007d` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:25.478253 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08826e27-8d87-4b75-851c-be6d026726f8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=067163b9757d46f7918f11725bdc007d)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:25.480817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08826e27-8d87-4b75-851c-be6d026726f8 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:25.490138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 899/1804] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:25 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:25.516762 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883abc37-9fdd-4ea6-adfb-de0434fbfb9d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:25.517988 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-883abc37-9fdd-4ea6-adfb-de0434fbfb9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:25.573376 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883abc37-9fdd-4ea6-adfb-de0434fbfb9d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:25.577564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35568d02-90fc-4d65-90c5-57190647e115 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:25.578164 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35568d02-90fc-4d65-90c5-57190647e115 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:25.578440 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35568d02-90fc-4d65-90c5-57190647e115 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:25.578743 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35568d02-90fc-4d65-90c5-57190647e115 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:25.587478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35568d02-90fc-4d65-90c5-57190647e115 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:25.593994 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35568d02-90fc-4d65-90c5-57190647e115 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:25.598024 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 900/1805] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:25 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:46:25.631931 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35568d02-90fc-4d65-90c5-57190647e115 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:25.642842 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35568d02-90fc-4d65-90c5-57190647e115 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:25.649984 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-08826e27-8d87-4b75-851c-be6d026726f8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '067163b9757d46f7918f11725bdc007d', 'request_id': 'req-08826e27-8d87-4b75-851c-be6d026726f8'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:25.656074 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-08826e27-8d87-4b75-851c-be6d026726f8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '067163b9757d46f7918f11725bdc007d', 'request_id': 'req-08826e27-8d87-4b75-851c-be6d026726f8'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:25.662309 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-08826e27-8d87-4b75-851c-be6d026726f8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '067163b9757d46f7918f11725bdc007d', 'request_id': 'req-08826e27-8d87-4b75-851c-be6d026726f8'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:25.703337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e09c804-9b4f-44a8-a2fa-5bd0e6193402 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:25.703992 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e09c804-9b4f-44a8-a2fa-5bd0e6193402 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:25.704268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e09c804-9b4f-44a8-a2fa-5bd0e6193402 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:25.704546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e09c804-9b4f-44a8-a2fa-5bd0e6193402 admin admin] PATH_INFO: `/v3/users/3477b16bb6ca47609fb29e21f52e08ab` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:25.705681 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e09c804-9b4f-44a8-a2fa-5bd0e6193402 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3477b16bb6ca47609fb29e21f52e08ab)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:25.707638 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e09c804-9b4f-44a8-a2fa-5bd0e6193402 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:25.745809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-08826e27-8d87-4b75-851c-be6d026726f8 admin admin] The token cache is being invalidate because project 067163b9757d46f7918f11725bdc007d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:46:25.746359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-08826e27-8d87-4b75-851c-be6d026726f8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:25.747036 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-08826e27-8d87-4b75-851c-be6d026726f8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:25.749189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 906/1806] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:46:25 2022] DELETE /identity/v3/projects/067163b9757d46f7918f11725bdc007d => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:25.752779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08826e27-8d87-4b75-851c-be6d026726f8 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:25.754196 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-08826e27-8d87-4b75-851c-be6d026726f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:25.768167 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08826e27-8d87-4b75-851c-be6d026726f8 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:25.774839 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb595cd-97bf-498c-8cdb-9bd75fc3226c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:25.775480 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb595cd-97bf-498c-8cdb-9bd75fc3226c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:25.775885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb595cd-97bf-498c-8cdb-9bd75fc3226c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:25.776167 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb595cd-97bf-498c-8cdb-9bd75fc3226c admin admin] PATH_INFO: `/v3/users/937e02a381fa41dd8ad94d3acc403d65` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:25.777285 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb595cd-97bf-498c-8cdb-9bd75fc3226c admin admin] RBAC: Authorizing `identity:delete_user(user_id=937e02a381fa41dd8ad94d3acc403d65)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:25.779425 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb595cd-97bf-498c-8cdb-9bd75fc3226c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:25.906684 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2e09c804-9b4f-44a8-a2fa-5bd0e6193402 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3477b16bb6ca47609fb29e21f52e08ab'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:25.907447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2e09c804-9b4f-44a8-a2fa-5bd0e6193402 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3477b16bb6ca47609fb29e21f52e08ab'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:25.908161 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2e09c804-9b4f-44a8-a2fa-5bd0e6193402 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3477b16bb6ca47609fb29e21f52e08ab'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:25.915691 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2e09c804-9b4f-44a8-a2fa-5bd0e6193402 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3477b16bb6ca47609fb29e21f52e08ab'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:25.951812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-feb595cd-97bf-498c-8cdb-9bd75fc3226c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '937e02a381fa41dd8ad94d3acc403d65'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:25.963741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-feb595cd-97bf-498c-8cdb-9bd75fc3226c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '937e02a381fa41dd8ad94d3acc403d65'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:25.964421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-feb595cd-97bf-498c-8cdb-9bd75fc3226c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '937e02a381fa41dd8ad94d3acc403d65'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:25.995708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2e09c804-9b4f-44a8-a2fa-5bd0e6193402 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3477b16bb6ca47609fb29e21f52e08ab'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:26.014386 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 901/1807] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:46:25 2022] DELETE /identity/v3/users/3477b16bb6ca47609fb29e21f52e08ab => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:26.018690 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb68a2a-006e-4937-8330-bef5ae79d061 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:26.018954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb68a2a-006e-4937-8330-bef5ae79d061 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:26.019232 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb68a2a-006e-4937-8330-bef5ae79d061 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:26.037179 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-feb68a2a-006e-4937-8330-bef5ae79d061 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:26.039909 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-feb68a2a-006e-4937-8330-bef5ae79d061 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:26.053846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-feb595cd-97bf-498c-8cdb-9bd75fc3226c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '937e02a381fa41dd8ad94d3acc403d65'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:26.061869 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-feb595cd-97bf-498c-8cdb-9bd75fc3226c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '937e02a381fa41dd8ad94d3acc403d65'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:26.068560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 907/1808] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:46:25 2022] DELETE /identity/v3/users/937e02a381fa41dd8ad94d3acc403d65 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:26.098235 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 902/1809] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:26.108645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb595cd-97bf-498c-8cdb-9bd75fc3226c admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:26.110201 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-feb595cd-97bf-498c-8cdb-9bd75fc3226c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:26.167760 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb595cd-97bf-498c-8cdb-9bd75fc3226c admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:26.172209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0dbc9b-3f9f-4329-b211-3acec48be2cc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:26.172830 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0dbc9b-3f9f-4329-b211-3acec48be2cc service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:26.173104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0dbc9b-3f9f-4329-b211-3acec48be2cc service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:26.173381 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0dbc9b-3f9f-4329-b211-3acec48be2cc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:26.248713 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d0dbc9b-3f9f-4329-b211-3acec48be2cc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:26.250639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d0dbc9b-3f9f-4329-b211-3acec48be2cc service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:26.263786 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 908/1810] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:46:26.527843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb68a2a-006e-4937-8330-bef5ae79d061 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:26.527843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-feb68a2a-006e-4937-8330-bef5ae79d061 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:26.558709 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb68a2a-006e-4937-8330-bef5ae79d061 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:26.564381 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153bfff1-af40-4a47-a120-67258a1f043b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:26.565094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153bfff1-af40-4a47-a120-67258a1f043b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:26.566987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153bfff1-af40-4a47-a120-67258a1f043b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:26.567298 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153bfff1-af40-4a47-a120-67258a1f043b admin admin] PATH_INFO: `/v3/projects/df9900d26732437b9f775eb11d6352ce` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:26.568531 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153bfff1-af40-4a47-a120-67258a1f043b admin admin] RBAC: Authorizing `identity:delete_project(project_id=df9900d26732437b9f775eb11d6352ce)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:26.570625 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153bfff1-af40-4a47-a120-67258a1f043b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:26.650535 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:INSERT=3,keystone:SELECT=90,keystone:DELETE=7 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:26.723211 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-153bfff1-af40-4a47-a120-67258a1f043b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df9900d26732437b9f775eb11d6352ce', 'request_id': 'req-153bfff1-af40-4a47-a120-67258a1f043b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:26.731725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-153bfff1-af40-4a47-a120-67258a1f043b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df9900d26732437b9f775eb11d6352ce', 'request_id': 'req-153bfff1-af40-4a47-a120-67258a1f043b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:26.739928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-153bfff1-af40-4a47-a120-67258a1f043b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df9900d26732437b9f775eb11d6352ce', 'request_id': 'req-153bfff1-af40-4a47-a120-67258a1f043b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:26.806349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-153bfff1-af40-4a47-a120-67258a1f043b admin admin] The token cache is being invalidate because project df9900d26732437b9f775eb11d6352ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:46:26.806886 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-153bfff1-af40-4a47-a120-67258a1f043b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:26.807573 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-153bfff1-af40-4a47-a120-67258a1f043b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:26.808438 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0dbc9b-3f9f-4329-b211-3acec48be2cc service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:26.809665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 903/1811] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:46:26 2022] DELETE /identity/v3/projects/df9900d26732437b9f775eb11d6352ce => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:26.810151 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-1d0dbc9b-3f9f-4329-b211-3acec48be2cc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:26.814759 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfc02b1-200b-44a0-84a4-fcf3041fb691 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:26.815882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfc02b1-200b-44a0-84a4-fcf3041fb691 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:26.816642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfc02b1-200b-44a0-84a4-fcf3041fb691 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:26.834866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0dbc9b-3f9f-4329-b211-3acec48be2cc service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:26.848945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-bdfc02b1-200b-44a0-84a4-fcf3041fb691 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:26.850280 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-bdfc02b1-200b-44a0-84a4-fcf3041fb691 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:26.912579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66049fc6-62ae-4719-a367-a8b1339b4324 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:26.913284 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66049fc6-62ae-4719-a367-a8b1339b4324 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:26.913661 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66049fc6-62ae-4719-a367-a8b1339b4324 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:26.914110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66049fc6-62ae-4719-a367-a8b1339b4324 admin admin] PATH_INFO: `/v3/projects/2cb74e9c881a43189117a06e1294e083` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:26.915283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66049fc6-62ae-4719-a367-a8b1339b4324 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2cb74e9c881a43189117a06e1294e083)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:26.917233 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66049fc6-62ae-4719-a367-a8b1339b4324 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:26.917387 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 904/1812] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:26.925757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e3956e-526a-46f0-9dbb-b7637a93ffea None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:26.926042 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e3956e-526a-46f0-9dbb-b7637a93ffea None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:26.926491 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e3956e-526a-46f0-9dbb-b7637a93ffea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:26.945785 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-15e3956e-526a-46f0-9dbb-b7637a93ffea None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:26.947261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-15e3956e-526a-46f0-9dbb-b7637a93ffea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:26.956947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 905/1813] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:26.961309 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e3956e-526a-46f0-9dbb-b7637a93ffea None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:26.971129 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e3956e-526a-46f0-9dbb-b7637a93ffea None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:26.975914 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5348ee43-6850-40b9-94ec-bfb9495cbb59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:26.976578 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5348ee43-6850-40b9-94ec-bfb9495cbb59 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:26.976914 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5348ee43-6850-40b9-94ec-bfb9495cbb59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:26.977195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5348ee43-6850-40b9-94ec-bfb9495cbb59 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:26.977842 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5348ee43-6850-40b9-94ec-bfb9495cbb59 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:46:26.978149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5348ee43-6850-40b9-94ec-bfb9495cbb59 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:26.980096 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5348ee43-6850-40b9-94ec-bfb9495cbb59 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:26.995261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 906/1814] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:46:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:46:26.999735 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5348ee43-6850-40b9-94ec-bfb9495cbb59 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:27.009873 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5348ee43-6850-40b9-94ec-bfb9495cbb59 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:27.016902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d72630-20ec-4e32-804d-5975dc43f63a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:27.019306 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d72630-20ec-4e32-804d-5975dc43f63a admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:27.019306 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d72630-20ec-4e32-804d-5975dc43f63a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:27.019306 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d72630-20ec-4e32-804d-5975dc43f63a admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:27.019306 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d72630-20ec-4e32-804d-5975dc43f63a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:27.028356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d72630-20ec-4e32-804d-5975dc43f63a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:27.039099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-66049fc6-62ae-4719-a367-a8b1339b4324 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2cb74e9c881a43189117a06e1294e083', 'request_id': 'req-66049fc6-62ae-4719-a367-a8b1339b4324'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:27.045269 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-66049fc6-62ae-4719-a367-a8b1339b4324 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2cb74e9c881a43189117a06e1294e083', 'request_id': 'req-66049fc6-62ae-4719-a367-a8b1339b4324'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:27.051744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-66049fc6-62ae-4719-a367-a8b1339b4324 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2cb74e9c881a43189117a06e1294e083', 'request_id': 'req-66049fc6-62ae-4719-a367-a8b1339b4324'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:27.151030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 907/1815] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:26 2022] POST /identity/v3/projects => generated 355 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:27.155015 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d72630-20ec-4e32-804d-5975dc43f63a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:27.187279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-66049fc6-62ae-4719-a367-a8b1339b4324 admin admin] The token cache is being invalidate because project 2cb74e9c881a43189117a06e1294e083 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:46:27.187805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-66049fc6-62ae-4719-a367-a8b1339b4324 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:27.188318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-66049fc6-62ae-4719-a367-a8b1339b4324 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:27.190132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 909/1816] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:46:26 2022] DELETE /identity/v3/projects/2cb74e9c881a43189117a06e1294e083 => generated 0 bytes in 382 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:27.193647 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66049fc6-62ae-4719-a367-a8b1339b4324 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:27.195725 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-66049fc6-62ae-4719-a367-a8b1339b4324 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:27.225727 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d72630-20ec-4e32-804d-5975dc43f63a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:27.230385 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7174ac93-3fde-4a99-88e8-4fd4da5fb0a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:27.231024 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7174ac93-3fde-4a99-88e8-4fd4da5fb0a9 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:27.231305 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7174ac93-3fde-4a99-88e8-4fd4da5fb0a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:27.231579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7174ac93-3fde-4a99-88e8-4fd4da5fb0a9 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:27.233247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7174ac93-3fde-4a99-88e8-4fd4da5fb0a9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:27.235865 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7174ac93-3fde-4a99-88e8-4fd4da5fb0a9 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:27.241150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:27.264447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66049fc6-62ae-4719-a367-a8b1339b4324 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:27.269032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a749b805-663b-4f83-a56f-04d2fc9f45b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:27.269658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a749b805-663b-4f83-a56f-04d2fc9f45b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:27.269935 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a749b805-663b-4f83-a56f-04d2fc9f45b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:27.270206 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a749b805-663b-4f83-a56f-04d2fc9f45b0 admin admin] PATH_INFO: `/v3/users/4256952c2eb74fdca049486509def40c` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:27.271336 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a749b805-663b-4f83-a56f-04d2fc9f45b0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4256952c2eb74fdca049486509def40c)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:27.273262 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a749b805-663b-4f83-a56f-04d2fc9f45b0 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:27.338402 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 908/1817] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:46:27 2022] POST /identity/v3/users => generated 278 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:46:27.342294 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7174ac93-3fde-4a99-88e8-4fd4da5fb0a9 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:27.344002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7174ac93-3fde-4a99-88e8-4fd4da5fb0a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:27.355947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7174ac93-3fde-4a99-88e8-4fd4da5fb0a9 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:27.361173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077b3980-a75a-4696-b61a-516b05debd11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:27.361773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077b3980-a75a-4696-b61a-516b05debd11 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:27.362108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077b3980-a75a-4696-b61a-516b05debd11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:27.362386 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077b3980-a75a-4696-b61a-516b05debd11 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:27.363061 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077b3980-a75a-4696-b61a-516b05debd11 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:27.364663 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077b3980-a75a-4696-b61a-516b05debd11 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:27.375480 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 909/1818] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:27 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:46:27.379224 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077b3980-a75a-4696-b61a-516b05debd11 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:27.389149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077b3980-a75a-4696-b61a-516b05debd11 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:27.394029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a504a0-28da-4745-8a96-f01e100bba48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:27.394680 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a504a0-28da-4745-8a96-f01e100bba48 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:27.394917 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a504a0-28da-4745-8a96-f01e100bba48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:27.395208 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a504a0-28da-4745-8a96-f01e100bba48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:27.395819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a504a0-28da-4745-8a96-f01e100bba48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:27.397398 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a504a0-28da-4745-8a96-f01e100bba48 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:27.408467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 910/1819] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:27 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:46:27.417888 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a504a0-28da-4745-8a96-f01e100bba48 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:27.437517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a504a0-28da-4745-8a96-f01e100bba48 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:27.442295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3023bc92-924a-48fd-9146-84c5f5a369c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:27.442977 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3023bc92-924a-48fd-9146-84c5f5a369c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:27.443266 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3023bc92-924a-48fd-9146-84c5f5a369c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:27.443544 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3023bc92-924a-48fd-9146-84c5f5a369c2 admin admin] PATH_INFO: `/v3/projects/e642241b30124f0cbf0fe8d9d67d8f47/users/851aecca45a64b058441375f78c4e2b1/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:27.445535 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a749b805-663b-4f83-a56f-04d2fc9f45b0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4256952c2eb74fdca049486509def40c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:27.458268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a749b805-663b-4f83-a56f-04d2fc9f45b0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4256952c2eb74fdca049486509def40c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:27.459018 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a749b805-663b-4f83-a56f-04d2fc9f45b0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4256952c2eb74fdca049486509def40c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:27.460850 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3023bc92-924a-48fd-9146-84c5f5a369c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e642241b30124f0cbf0fe8d9d67d8f47, user_id=851aecca45a64b058441375f78c4e2b1, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:27.463223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3023bc92-924a-48fd-9146-84c5f5a369c2 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:27.561971 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 911/1820] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:27 2022] PUT /identity/v3/projects/e642241b30124f0cbf0fe8d9d67d8f47/users/851aecca45a64b058441375f78c4e2b1/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:27.573371 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da059dd8-42fc-4818-bf2d-a8cb4884234b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:27.574197 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da059dd8-42fc-4818-bf2d-a8cb4884234b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:27.574197 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da059dd8-42fc-4818-bf2d-a8cb4884234b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:27.585558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:27.586331 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a749b805-663b-4f83-a56f-04d2fc9f45b0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4256952c2eb74fdca049486509def40c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:27.595079 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a749b805-663b-4f83-a56f-04d2fc9f45b0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4256952c2eb74fdca049486509def40c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:27.603836 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 910/1821] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:46:27 2022] DELETE /identity/v3/users/4256952c2eb74fdca049486509def40c => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:27.610656 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-da059dd8-42fc-4818-bf2d-a8cb4884234b None None] MFA Rules not processed for user `851aecca45a64b058441375f78c4e2b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:27.611947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-da059dd8-42fc-4818-bf2d-a8cb4884234b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:27.680705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 912/1822] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:27 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:27.685501 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a749b805-663b-4f83-a56f-04d2fc9f45b0 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:27.687602 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a749b805-663b-4f83-a56f-04d2fc9f45b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:27.754360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a749b805-663b-4f83-a56f-04d2fc9f45b0 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:27.761063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81efb9c5-dc35-424c-8395-b3475774f6e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:27.761676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81efb9c5-dc35-424c-8395-b3475774f6e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:27.763170 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81efb9c5-dc35-424c-8395-b3475774f6e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:27.763170 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81efb9c5-dc35-424c-8395-b3475774f6e1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:27.763170 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81efb9c5-dc35-424c-8395-b3475774f6e1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:27.764964 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81efb9c5-dc35-424c-8395-b3475774f6e1 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:27.792527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:27.873538 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 911/1823] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:27 2022] POST /identity/v3/projects => generated 357 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:27.878242 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da059dd8-42fc-4818-bf2d-a8cb4884234b None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:27.889624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da059dd8-42fc-4818-bf2d-a8cb4884234b None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:27.952549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d37f22-abd5-41c7-aaca-e2411ef98952 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:27.953268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d37f22-abd5-41c7-aaca-e2411ef98952 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:27.953619 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d37f22-abd5-41c7-aaca-e2411ef98952 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:27.953960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d37f22-abd5-41c7-aaca-e2411ef98952 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:27.954696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d37f22-abd5-41c7-aaca-e2411ef98952 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:27.956912 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d37f22-abd5-41c7-aaca-e2411ef98952 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:28.061814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81efb9c5-dc35-424c-8395-b3475774f6e1 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:28.063825 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-81efb9c5-dc35-424c-8395-b3475774f6e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:28.091037 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81efb9c5-dc35-424c-8395-b3475774f6e1 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:28.095742 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 913/1824] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:46:27 2022] POST /identity/v3/users => generated 285 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:28.100160 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03208d56-0476-4986-990a-fa4b25d6b69b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:28.100581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03208d56-0476-4986-990a-fa4b25d6b69b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:28.100581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03208d56-0476-4986-990a-fa4b25d6b69b admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:28.100745 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03208d56-0476-4986-990a-fa4b25d6b69b admin admin] PATH_INFO: `/v3/projects/0d17d9f4c2d442f08f77adb7703aa309` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:28.101405 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d37f22-abd5-41c7-aaca-e2411ef98952 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:28.103539 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03208d56-0476-4986-990a-fa4b25d6b69b admin admin] RBAC: Authorizing `identity:delete_project(project_id=0d17d9f4c2d442f08f77adb7703aa309)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:28.110961 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03208d56-0476-4986-990a-fa4b25d6b69b admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:28.112939 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d37f22-abd5-41c7-aaca-e2411ef98952 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:28.118832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdfd6e9-bc2d-457f-a801-c8cd2a6fbe0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:28.119638 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdfd6e9-bc2d-457f-a801-c8cd2a6fbe0c admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:28.119929 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdfd6e9-bc2d-457f-a801-c8cd2a6fbe0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:28.120211 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdfd6e9-bc2d-457f-a801-c8cd2a6fbe0c admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:28.120864 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fdfd6e9-bc2d-457f-a801-c8cd2a6fbe0c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:28.122879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fdfd6e9-bc2d-457f-a801-c8cd2a6fbe0c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:28.134038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 914/1825] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:28 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:28.139311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdfd6e9-bc2d-457f-a801-c8cd2a6fbe0c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:28.153311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdfd6e9-bc2d-457f-a801-c8cd2a6fbe0c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:28.158047 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f663b238-d967-4d34-8327-7de42f4fb074 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:28.158802 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f663b238-d967-4d34-8327-7de42f4fb074 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:28.159088 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f663b238-d967-4d34-8327-7de42f4fb074 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:28.159365 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f663b238-d967-4d34-8327-7de42f4fb074 admin admin] PATH_INFO: `/v3/projects/48eb6d0c3f1f4cf98dd5707729fcf461/users/7eb49e88af3e422581776a81236f31af/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:28.174286 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f663b238-d967-4d34-8327-7de42f4fb074 admin admin] RBAC: Authorizing `identity:create_grant(project_id=48eb6d0c3f1f4cf98dd5707729fcf461, user_id=7eb49e88af3e422581776a81236f31af, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:28.177042 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f663b238-d967-4d34-8327-7de42f4fb074 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:28.288870 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 915/1826] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:28 2022] PUT /identity/v3/projects/48eb6d0c3f1f4cf98dd5707729fcf461/users/7eb49e88af3e422581776a81236f31af/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:28.291937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f663b238-d967-4d34-8327-7de42f4fb074 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:28.300757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-03208d56-0476-4986-990a-fa4b25d6b69b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0d17d9f4c2d442f08f77adb7703aa309', 'request_id': 'req-03208d56-0476-4986-990a-fa4b25d6b69b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:28.307187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-03208d56-0476-4986-990a-fa4b25d6b69b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0d17d9f4c2d442f08f77adb7703aa309', 'request_id': 'req-03208d56-0476-4986-990a-fa4b25d6b69b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:28.314752 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-03208d56-0476-4986-990a-fa4b25d6b69b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0d17d9f4c2d442f08f77adb7703aa309', 'request_id': 'req-03208d56-0476-4986-990a-fa4b25d6b69b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:28.355064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f663b238-d967-4d34-8327-7de42f4fb074 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:28.409524 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-03208d56-0476-4986-990a-fa4b25d6b69b admin admin] The token cache is being invalidate because project 0d17d9f4c2d442f08f77adb7703aa309 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:46:28.409736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-03208d56-0476-4986-990a-fa4b25d6b69b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:28.410393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-03208d56-0476-4986-990a-fa4b25d6b69b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:28.412248 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 912/1827] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:46:28 2022] DELETE /identity/v3/projects/0d17d9f4c2d442f08f77adb7703aa309 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:28.412356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4d8de5-8d12-4e30-b91e-db1eb61235a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:28.412737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4d8de5-8d12-4e30-b91e-db1eb61235a6 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:28.412967 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4d8de5-8d12-4e30-b91e-db1eb61235a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:28.413243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4d8de5-8d12-4e30-b91e-db1eb61235a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:28.413879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4d8de5-8d12-4e30-b91e-db1eb61235a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:28.415555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4d8de5-8d12-4e30-b91e-db1eb61235a6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:28.416791 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cc768b-3a55-46c4-ba21-58ac2386f3ed None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:28.416977 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cc768b-3a55-46c4-ba21-58ac2386f3ed None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:28.417307 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cc768b-3a55-46c4-ba21-58ac2386f3ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:28.427702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 916/1828] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:28 2022] GET /identity/v3/roles => generated 1420 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:46:28.432039 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4d8de5-8d12-4e30-b91e-db1eb61235a6 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:28.436793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8b4d8de5-8d12-4e30-b91e-db1eb61235a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:28.442288 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-72cc768b-3a55-46c4-ba21-58ac2386f3ed None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:28.443578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-72cc768b-3a55-46c4-ba21-58ac2386f3ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:28.447187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4d8de5-8d12-4e30-b91e-db1eb61235a6 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:28.452224 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aba3319-8a02-4f69-8f36-3a874b71d5c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:28.453019 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aba3319-8a02-4f69-8f36-3a874b71d5c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:28.453293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aba3319-8a02-4f69-8f36-3a874b71d5c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:28.453566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aba3319-8a02-4f69-8f36-3a874b71d5c4 admin admin] PATH_INFO: `/v3/projects/48eb6d0c3f1f4cf98dd5707729fcf461/users/7eb49e88af3e422581776a81236f31af/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:28.455568 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aba3319-8a02-4f69-8f36-3a874b71d5c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=48eb6d0c3f1f4cf98dd5707729fcf461, user_id=7eb49e88af3e422581776a81236f31af, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:28.458648 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aba3319-8a02-4f69-8f36-3a874b71d5c4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:28.460026 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 913/1829] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:28.468350 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b8f623-d9d8-4cfa-a638-1140ef443ba0 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:28.468637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b8f623-d9d8-4cfa-a638-1140ef443ba0 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:28.468964 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b8f623-d9d8-4cfa-a638-1140ef443ba0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:28.475166 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 917/1830] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:28 2022] PUT /identity/v3/projects/48eb6d0c3f1f4cf98dd5707729fcf461/users/7eb49e88af3e422581776a81236f31af/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:28.491429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61472605-b885-4ede-8085-70a66400cc74 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:28.491878 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61472605-b885-4ede-8085-70a66400cc74 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:28.492323 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61472605-b885-4ede-8085-70a66400cc74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:28.494408 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-b8b8f623-d9d8-4cfa-a638-1140ef443ba0 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:28.495674 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b8b8f623-d9d8-4cfa-a638-1140ef443ba0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:28.527163 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-61472605-b885-4ede-8085-70a66400cc74 None None] MFA Rules not processed for user `7eb49e88af3e422581776a81236f31af`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:28.527163 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-61472605-b885-4ede-8085-70a66400cc74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:28.546736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:28.569566 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 914/1831] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:28.574195 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b8f623-d9d8-4cfa-a638-1140ef443ba0 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:28.582842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b8f623-d9d8-4cfa-a638-1140ef443ba0 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:28.587610 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614ec00a-ec0b-4157-b7c9-e2af4ee3def2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:28.588203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614ec00a-ec0b-4157-b7c9-e2af4ee3def2 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:28.588482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614ec00a-ec0b-4157-b7c9-e2af4ee3def2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:28.588757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614ec00a-ec0b-4157-b7c9-e2af4ee3def2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:28.589394 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614ec00a-ec0b-4157-b7c9-e2af4ee3def2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:46:28.589933 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614ec00a-ec0b-4157-b7c9-e2af4ee3def2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:28.592121 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614ec00a-ec0b-4157-b7c9-e2af4ee3def2 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:28.600150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 918/1832] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:28 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:28.604289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 915/1833] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:46:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:46:28.608799 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61472605-b885-4ede-8085-70a66400cc74 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:28.614422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614ec00a-ec0b-4157-b7c9-e2af4ee3def2 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:28.616130 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-614ec00a-ec0b-4157-b7c9-e2af4ee3def2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:28.619631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61472605-b885-4ede-8085-70a66400cc74 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:28.625590 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267afd4f-c06a-4f5d-b99e-6fe7e3f6e4c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:28.626200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267afd4f-c06a-4f5d-b99e-6fe7e3f6e4c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:28.626479 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267afd4f-c06a-4f5d-b99e-6fe7e3f6e4c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:28.626806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267afd4f-c06a-4f5d-b99e-6fe7e3f6e4c4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:28.627432 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267afd4f-c06a-4f5d-b99e-6fe7e3f6e4c4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:28.629321 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267afd4f-c06a-4f5d-b99e-6fe7e3f6e4c4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:28.655309 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:28.674857 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614ec00a-ec0b-4157-b7c9-e2af4ee3def2 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:28.678509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c8d8b5-a521-4768-bc1c-be4cd0f77cd2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:28.679145 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c8d8b5-a521-4768-bc1c-be4cd0f77cd2 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:28.679430 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c8d8b5-a521-4768-bc1c-be4cd0f77cd2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:28.679782 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c8d8b5-a521-4768-bc1c-be4cd0f77cd2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:28.689115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c8d8b5-a521-4768-bc1c-be4cd0f77cd2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:28.690832 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c8d8b5-a521-4768-bc1c-be4cd0f77cd2 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:28.701161 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 916/1834] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:28 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:46:28.721517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 919/1835] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:28 2022] POST /identity/v3/projects => generated 365 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:28.722739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c8d8b5-a521-4768-bc1c-be4cd0f77cd2 service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:28.733046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c8d8b5-a521-4768-bc1c-be4cd0f77cd2 service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:28.789180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06f4952-19e7-4031-8225-ce9a083ee999 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:28.789979 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06f4952-19e7-4031-8225-ce9a083ee999 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:28.790262 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06f4952-19e7-4031-8225-ce9a083ee999 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:28.790537 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06f4952-19e7-4031-8225-ce9a083ee999 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:28.791209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06f4952-19e7-4031-8225-ce9a083ee999 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:28.793151 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06f4952-19e7-4031-8225-ce9a083ee999 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:28.883053 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 917/1836] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:46:28 2022] POST /identity/v3/users => generated 283 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:46:28.886859 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267afd4f-c06a-4f5d-b99e-6fe7e3f6e4c4 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:28.897333 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267afd4f-c06a-4f5d-b99e-6fe7e3f6e4c4 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:28.902266 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c9c4f2-572a-475b-ba5f-95495250eed4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:28.902938 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c9c4f2-572a-475b-ba5f-95495250eed4 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:28.903219 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c9c4f2-572a-475b-ba5f-95495250eed4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:28.904742 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c9c4f2-572a-475b-ba5f-95495250eed4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:28.907116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c9c4f2-572a-475b-ba5f-95495250eed4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:28.909750 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c9c4f2-572a-475b-ba5f-95495250eed4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:28.921657 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 920/1837] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:28 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:28.925521 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06f4952-19e7-4031-8225-ce9a083ee999 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:28.925769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c9c4f2-572a-475b-ba5f-95495250eed4 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:28.935163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06f4952-19e7-4031-8225-ce9a083ee999 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:28.939718 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4b5e41-7ff0-49be-93ef-193f1fdefb61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:28.940308 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4b5e41-7ff0-49be-93ef-193f1fdefb61 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:28.940585 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4b5e41-7ff0-49be-93ef-193f1fdefb61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:28.940856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4b5e41-7ff0-49be-93ef-193f1fdefb61 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:28.941487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4b5e41-7ff0-49be-93ef-193f1fdefb61 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:28.943115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4b5e41-7ff0-49be-93ef-193f1fdefb61 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:28.953239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 918/1838] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:28 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:28.957229 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4b5e41-7ff0-49be-93ef-193f1fdefb61 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:28.967394 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4b5e41-7ff0-49be-93ef-193f1fdefb61 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:28.972085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ea0db5-5c93-488a-a5ed-e064e7e1eb62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:28.972730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ea0db5-5c93-488a-a5ed-e064e7e1eb62 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:28.973016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ea0db5-5c93-488a-a5ed-e064e7e1eb62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:28.973294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ea0db5-5c93-488a-a5ed-e064e7e1eb62 admin admin] PATH_INFO: `/v3/projects/6d70aed35ae04e5fb792cbf92ac67c71/users/b907fec01282484f87d4e0091ae84ba4/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:28.990077 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ea0db5-5c93-488a-a5ed-e064e7e1eb62 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d70aed35ae04e5fb792cbf92ac67c71, user_id=b907fec01282484f87d4e0091ae84ba4, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:28.991871 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c9c4f2-572a-475b-ba5f-95495250eed4 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:28.993469 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ea0db5-5c93-488a-a5ed-e064e7e1eb62 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:28.995569 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7417ac94-1ac4-4932-aff7-a1823d1e01b0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:28.996584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7417ac94-1ac4-4932-aff7-a1823d1e01b0 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:28.996866 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7417ac94-1ac4-4932-aff7-a1823d1e01b0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:28.997159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7417ac94-1ac4-4932-aff7-a1823d1e01b0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:28.999438 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7417ac94-1ac4-4932-aff7-a1823d1e01b0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:29.064992 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 919/1839] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:28 2022] PUT /identity/v3/projects/6d70aed35ae04e5fb792cbf92ac67c71/users/b907fec01282484f87d4e0091ae84ba4/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:29.074156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7417ac94-1ac4-4932-aff7-a1823d1e01b0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:29.074156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7417ac94-1ac4-4932-aff7-a1823d1e01b0 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:29.077771 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521cb580-5f19-44cd-af53-3c963795b2e8 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:29.078226 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521cb580-5f19-44cd-af53-3c963795b2e8 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:29.078672 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521cb580-5f19-44cd-af53-3c963795b2e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:29.092835 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 921/1840] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:28 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:46:29.117077 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-521cb580-5f19-44cd-af53-3c963795b2e8 None None] MFA Rules not processed for user `b907fec01282484f87d4e0091ae84ba4`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:29.118369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-521cb580-5f19-44cd-af53-3c963795b2e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:29.183877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:29.203831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 920/1841] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:29 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:29.208616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7417ac94-1ac4-4932-aff7-a1823d1e01b0 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:29.222183 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7417ac94-1ac4-4932-aff7-a1823d1e01b0 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:29.288840 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0674355c-b791-42e9-9e39-90af13b3e2c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:29.289737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0674355c-b791-42e9-9e39-90af13b3e2c9 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:29.290243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0674355c-b791-42e9-9e39-90af13b3e2c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:29.290730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0674355c-b791-42e9-9e39-90af13b3e2c9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:29.292039 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0674355c-b791-42e9-9e39-90af13b3e2c9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:29.294336 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0674355c-b791-42e9-9e39-90af13b3e2c9 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:29.394775 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 922/1842] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:29 2022] POST /identity/v3/projects => generated 365 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:29.406541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521cb580-5f19-44cd-af53-3c963795b2e8 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:29.417530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521cb580-5f19-44cd-af53-3c963795b2e8 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:29.473411 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8b0e88-dee9-4d52-a224-6c0939ac70d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:29.474028 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8b0e88-dee9-4d52-a224-6c0939ac70d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:29.474313 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8b0e88-dee9-4d52-a224-6c0939ac70d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:29.474617 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8b0e88-dee9-4d52-a224-6c0939ac70d7 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:29.475234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8b0e88-dee9-4d52-a224-6c0939ac70d7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:29.477295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8b0e88-dee9-4d52-a224-6c0939ac70d7 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:29.483688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:29.556594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 921/1843] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:46:29 2022] POST /identity/v3/users => generated 289 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:29.563378 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0674355c-b791-42e9-9e39-90af13b3e2c9 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:29.573805 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0674355c-b791-42e9-9e39-90af13b3e2c9 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:29.578472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c50ae0d-bfe0-4d0d-8655-6313cdbcba8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:29.579113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c50ae0d-bfe0-4d0d-8655-6313cdbcba8c admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:29.579393 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c50ae0d-bfe0-4d0d-8655-6313cdbcba8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:29.579675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c50ae0d-bfe0-4d0d-8655-6313cdbcba8c admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:29.580317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c50ae0d-bfe0-4d0d-8655-6313cdbcba8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:29.581907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c50ae0d-bfe0-4d0d-8655-6313cdbcba8c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:29.592530 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 923/1844] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:29 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:29.596743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8b0e88-dee9-4d52-a224-6c0939ac70d7 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:29.608564 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8b0e88-dee9-4d52-a224-6c0939ac70d7 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:29.613367 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9468da-ae24-4a1d-afdb-0cf0fe6aa696 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:29.614019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9468da-ae24-4a1d-afdb-0cf0fe6aa696 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:29.614298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9468da-ae24-4a1d-afdb-0cf0fe6aa696 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:29.614569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9468da-ae24-4a1d-afdb-0cf0fe6aa696 admin admin] PATH_INFO: `/v3/projects/22ab318fbcb645bdafd7522089e44cdb/users/ad52a20c66db493983ca762d17dc7cec/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:29.631198 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9468da-ae24-4a1d-afdb-0cf0fe6aa696 admin admin] RBAC: Authorizing `identity:create_grant(project_id=22ab318fbcb645bdafd7522089e44cdb, user_id=ad52a20c66db493983ca762d17dc7cec, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:29.633886 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9468da-ae24-4a1d-afdb-0cf0fe6aa696 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:29.743910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 922/1845] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:29 2022] PUT /identity/v3/projects/22ab318fbcb645bdafd7522089e44cdb/users/ad52a20c66db493983ca762d17dc7cec/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:29.748820 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c50ae0d-bfe0-4d0d-8655-6313cdbcba8c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:29.758540 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c50ae0d-bfe0-4d0d-8655-6313cdbcba8c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:29.812778 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cb42cc-38fb-488e-8162-2d41cd7933ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:29.813388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cb42cc-38fb-488e-8162-2d41cd7933ae admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:29.813667 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cb42cc-38fb-488e-8162-2d41cd7933ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:29.813940 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cb42cc-38fb-488e-8162-2d41cd7933ae admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:29.814574 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12cb42cc-38fb-488e-8162-2d41cd7933ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:29.816187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12cb42cc-38fb-488e-8162-2d41cd7933ae admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:29.829675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 924/1846] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:29 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:29.829978 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:29.831409 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9468da-ae24-4a1d-afdb-0cf0fe6aa696 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:29.842561 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9468da-ae24-4a1d-afdb-0cf0fe6aa696 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:29.847310 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b206fca-e8b3-4d41-a975-e722cb7276e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:29.847959 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b206fca-e8b3-4d41-a975-e722cb7276e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:29.848238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b206fca-e8b3-4d41-a975-e722cb7276e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:29.848527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b206fca-e8b3-4d41-a975-e722cb7276e3 admin admin] PATH_INFO: `/v3/projects/22ab318fbcb645bdafd7522089e44cdb/users/ad52a20c66db493983ca762d17dc7cec/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:29.850392 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b206fca-e8b3-4d41-a975-e722cb7276e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=22ab318fbcb645bdafd7522089e44cdb, user_id=ad52a20c66db493983ca762d17dc7cec, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:29.853098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b206fca-e8b3-4d41-a975-e722cb7276e3 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:29.864306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 923/1847] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:29 2022] PUT /identity/v3/projects/22ab318fbcb645bdafd7522089e44cdb/users/ad52a20c66db493983ca762d17dc7cec/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:29.876183 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e951ad91-03ca-43ba-b1e2-2e163701c4c0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:29.878285 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e951ad91-03ca-43ba-b1e2-2e163701c4c0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:29.878285 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e951ad91-03ca-43ba-b1e2-2e163701c4c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:29.905782 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e951ad91-03ca-43ba-b1e2-2e163701c4c0 None None] MFA Rules not processed for user `ad52a20c66db493983ca762d17dc7cec`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:29.907122 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e951ad91-03ca-43ba-b1e2-2e163701c4c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:29.952114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:29.978249 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 925/1848] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:29 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:29.989867 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b206fca-e8b3-4d41-a975-e722cb7276e3 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:30.048731 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b206fca-e8b3-4d41-a975-e722cb7276e3 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:30.052450 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6e4da1-02fa-47c5-be2f-9b7bb2ff9b96 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:30.053056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6e4da1-02fa-47c5-be2f-9b7bb2ff9b96 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:30.053339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6e4da1-02fa-47c5-be2f-9b7bb2ff9b96 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:30.053613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6e4da1-02fa-47c5-be2f-9b7bb2ff9b96 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:30.062415 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6e4da1-02fa-47c5-be2f-9b7bb2ff9b96 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:30.063887 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6e4da1-02fa-47c5-be2f-9b7bb2ff9b96 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:30.073253 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 924/1849] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:29 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:46:30.333450 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e951ad91-03ca-43ba-b1e2-2e163701c4c0 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:30.335674 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e951ad91-03ca-43ba-b1e2-2e163701c4c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:30.349224 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e951ad91-03ca-43ba-b1e2-2e163701c4c0 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:30.353120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3329719e-aa97-4a75-95ff-8e155478ec64 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:30.354027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3329719e-aa97-4a75-95ff-8e155478ec64 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:30.354528 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3329719e-aa97-4a75-95ff-8e155478ec64 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:30.354940 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3329719e-aa97-4a75-95ff-8e155478ec64 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:30.420399 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3329719e-aa97-4a75-95ff-8e155478ec64 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:30.421993 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3329719e-aa97-4a75-95ff-8e155478ec64 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:30.431229 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 926/1850] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:30 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:46:30.694380 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:30.746112 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:31.447212 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:32.265422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:41.041578 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:42.352298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=29,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:42.502355 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd5a722-e6f5-4eaa-be72-0f4f6c52f591 None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:42.502671 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd5a722-e6f5-4eaa-be72-0f4f6c52f591 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:42.502951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd5a722-e6f5-4eaa-be72-0f4f6c52f591 None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:42.504818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 925/1851] 10.222.0.39 () {54 vars in 921 bytes} [Fri Sep 30 09:46:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:46:42.512895 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b40d25-05a0-4a3c-be78-5928491d28fb None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:42.513447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b40d25-05a0-4a3c-be78-5928491d28fb None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:42.513447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b40d25-05a0-4a3c-be78-5928491d28fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:42.539092 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c4b40d25-05a0-4a3c-be78-5928491d28fb None None] MFA Rules not processed for user `7ff68fb87700423ab53c90520a6bd3b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:42.540411 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c4b40d25-05a0-4a3c-be78-5928491d28fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:42.617076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 927/1852] 10.222.0.39 () {58 vars in 1018 bytes} [Fri Sep 30 09:46:42 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:42.642146 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd5a722-e6f5-4eaa-be72-0f4f6c52f591 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:42.644097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-9bd5a722-e6f5-4eaa-be72-0f4f6c52f591 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:42.667924 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd5a722-e6f5-4eaa-be72-0f4f6c52f591 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:42.675957 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c7a3e6-5593-499e-bebf-5427903c19c7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:42.677419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c7a3e6-5593-499e-bebf-5427903c19c7 service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:42.677879 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c7a3e6-5593-499e-bebf-5427903c19c7 service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:42.678345 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c7a3e6-5593-499e-bebf-5427903c19c7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:42.692951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c7a3e6-5593-499e-bebf-5427903c19c7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:42.695438 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c7a3e6-5593-499e-bebf-5427903c19c7 service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:42.721652 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 926/1853] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:46:49.458878 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b40d25-05a0-4a3c-be78-5928491d28fb None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:49.560360 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b40d25-05a0-4a3c-be78-5928491d28fb None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:49.568828 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7fa3d8-802b-4005-a13a-1968c79568bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:49.569907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7fa3d8-802b-4005-a13a-1968c79568bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:49.570386 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7fa3d8-802b-4005-a13a-1968c79568bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:49.570908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7fa3d8-802b-4005-a13a-1968c79568bb admin admin] PATH_INFO: `/v3/users/b907fec01282484f87d4e0091ae84ba4` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:49.572529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7fa3d8-802b-4005-a13a-1968c79568bb admin admin] RBAC: Authorizing `identity:delete_user(user_id=b907fec01282484f87d4e0091ae84ba4)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:49.576082 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7fa3d8-802b-4005-a13a-1968c79568bb admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:49.753505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c7a3e6-5593-499e-bebf-5427903c19c7 service nova] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:49.755070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-43c7a3e6-5593-499e-bebf-5427903c19c7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:49.771428 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c7a3e6-5593-499e-bebf-5427903c19c7 service nova] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:49.778668 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df015912-be9f-4f09-94f7-16c04d8e3d8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:49.780477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df015912-be9f-4f09-94f7-16c04d8e3d8f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:49.780891 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df015912-be9f-4f09-94f7-16c04d8e3d8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:49.781600 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df015912-be9f-4f09-94f7-16c04d8e3d8f admin admin] PATH_INFO: `/v3/users/30170cffd964441a89299422c7738a8b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:49.782791 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5d7fa3d8-802b-4005-a13a-1968c79568bb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b907fec01282484f87d4e0091ae84ba4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:49.782918 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df015912-be9f-4f09-94f7-16c04d8e3d8f admin admin] RBAC: Authorizing `identity:delete_user(user_id=30170cffd964441a89299422c7738a8b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:49.783474 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5d7fa3d8-802b-4005-a13a-1968c79568bb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b907fec01282484f87d4e0091ae84ba4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:49.784129 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5d7fa3d8-802b-4005-a13a-1968c79568bb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b907fec01282484f87d4e0091ae84ba4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:49.784912 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df015912-be9f-4f09-94f7-16c04d8e3d8f admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:49.793421 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5d7fa3d8-802b-4005-a13a-1968c79568bb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b907fec01282484f87d4e0091ae84ba4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:49.887787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5d7fa3d8-802b-4005-a13a-1968c79568bb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b907fec01282484f87d4e0091ae84ba4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:49.918622 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 928/1854] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:46:49 2022] DELETE /identity/v3/users/b907fec01282484f87d4e0091ae84ba4 => generated 0 bytes in 460 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:49.923954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c7c500-048d-4135-8ec5-356841184311 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:49.926108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c7c500-048d-4135-8ec5-356841184311 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:49.926108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c7c500-048d-4135-8ec5-356841184311 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:49.943828 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e9c7c500-048d-4135-8ec5-356841184311 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:49.945092 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e9c7c500-048d-4135-8ec5-356841184311 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:49.950224 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:49.987568 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-df015912-be9f-4f09-94f7-16c04d8e3d8f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '30170cffd964441a89299422c7738a8b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:50.000469 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-df015912-be9f-4f09-94f7-16c04d8e3d8f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '30170cffd964441a89299422c7738a8b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:50.001065 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-df015912-be9f-4f09-94f7-16c04d8e3d8f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '30170cffd964441a89299422c7738a8b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:50.007220 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 929/1855] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:50.017507 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c7c500-048d-4135-8ec5-356841184311 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:50.019049 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e9c7c500-048d-4135-8ec5-356841184311 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:50.072697 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c7c500-048d-4135-8ec5-356841184311 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:50.076322 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d95fb82-c36c-4363-bcf4-d7793f77d0fa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:50.077475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d95fb82-c36c-4363-bcf4-d7793f77d0fa service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:50.077776 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d95fb82-c36c-4363-bcf4-d7793f77d0fa service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:50.078065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d95fb82-c36c-4363-bcf4-d7793f77d0fa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:50.087036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d95fb82-c36c-4363-bcf4-d7793f77d0fa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:50.088455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d95fb82-c36c-4363-bcf4-d7793f77d0fa service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:50.093218 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-df015912-be9f-4f09-94f7-16c04d8e3d8f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '30170cffd964441a89299422c7738a8b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:50.105688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 930/1856] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:46:50.105820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-df015912-be9f-4f09-94f7-16c04d8e3d8f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '30170cffd964441a89299422c7738a8b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:50.109460 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 927/1857] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:46:49 2022] DELETE /identity/v3/users/30170cffd964441a89299422c7738a8b => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:50.114365 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7f4443-b01f-479b-a91d-6a1da904fb72 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:50.114365 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7f4443-b01f-479b-a91d-6a1da904fb72 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:50.114581 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7f4443-b01f-479b-a91d-6a1da904fb72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:50.134351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-5f7f4443-b01f-479b-a91d-6a1da904fb72 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:50.135621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5f7f4443-b01f-479b-a91d-6a1da904fb72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:50.203008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 931/1858] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:50.214784 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df015912-be9f-4f09-94f7-16c04d8e3d8f admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:50.290968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df015912-be9f-4f09-94f7-16c04d8e3d8f admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:50.307268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e6f18d-df2c-489b-902c-34511096c15a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:50.308002 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e6f18d-df2c-489b-902c-34511096c15a service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:50.308292 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e6f18d-df2c-489b-902c-34511096c15a service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:50.308627 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e6f18d-df2c-489b-902c-34511096c15a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:50.321731 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e6f18d-df2c-489b-902c-34511096c15a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:50.323876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e6f18d-df2c-489b-902c-34511096c15a service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:50.337158 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 928/1859] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:46:50.716108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7f4443-b01f-479b-a91d-6a1da904fb72 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:50.717569 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5f7f4443-b01f-479b-a91d-6a1da904fb72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:50.729514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7f4443-b01f-479b-a91d-6a1da904fb72 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:50.734728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a04371-b2ff-4d98-9103-c364b6ad5cf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:50.735373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a04371-b2ff-4d98-9103-c364b6ad5cf8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:50.735648 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a04371-b2ff-4d98-9103-c364b6ad5cf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:50.735919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a04371-b2ff-4d98-9103-c364b6ad5cf8 admin admin] PATH_INFO: `/v3/projects/6d70aed35ae04e5fb792cbf92ac67c71` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:50.737058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a04371-b2ff-4d98-9103-c364b6ad5cf8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6d70aed35ae04e5fb792cbf92ac67c71)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:50.739003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a04371-b2ff-4d98-9103-c364b6ad5cf8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:50.854063 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-95a04371-b2ff-4d98-9103-c364b6ad5cf8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6d70aed35ae04e5fb792cbf92ac67c71', 'request_id': 'req-95a04371-b2ff-4d98-9103-c364b6ad5cf8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:50.855425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:50.865180 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-95a04371-b2ff-4d98-9103-c364b6ad5cf8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6d70aed35ae04e5fb792cbf92ac67c71', 'request_id': 'req-95a04371-b2ff-4d98-9103-c364b6ad5cf8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:50.879161 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-95a04371-b2ff-4d98-9103-c364b6ad5cf8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6d70aed35ae04e5fb792cbf92ac67c71', 'request_id': 'req-95a04371-b2ff-4d98-9103-c364b6ad5cf8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:50.978251 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-95a04371-b2ff-4d98-9103-c364b6ad5cf8 admin admin] The token cache is being invalidate because project 6d70aed35ae04e5fb792cbf92ac67c71 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:46:50.978541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-95a04371-b2ff-4d98-9103-c364b6ad5cf8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:50.982398 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-95a04371-b2ff-4d98-9103-c364b6ad5cf8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:50.982398 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 932/1860] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:46:50 2022] DELETE /identity/v3/projects/6d70aed35ae04e5fb792cbf92ac67c71 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:50.985123 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e6f18d-df2c-489b-902c-34511096c15a service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:50.995081 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e6f18d-df2c-489b-902c-34511096c15a service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:51.015805 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a04371-b2ff-4d98-9103-c364b6ad5cf8 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:51.019774 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-95a04371-b2ff-4d98-9103-c364b6ad5cf8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:51.090201 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4f6765-36ce-4137-b9e1-e08d67ff9aed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:51.090893 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4f6765-36ce-4137-b9e1-e08d67ff9aed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:51.091174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4f6765-36ce-4137-b9e1-e08d67ff9aed admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:51.092585 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4f6765-36ce-4137-b9e1-e08d67ff9aed admin admin] PATH_INFO: `/v3/users/ad52a20c66db493983ca762d17dc7cec` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:51.093810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4f6765-36ce-4137-b9e1-e08d67ff9aed admin admin] RBAC: Authorizing `identity:delete_user(user_id=ad52a20c66db493983ca762d17dc7cec)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:51.103115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4f6765-36ce-4137-b9e1-e08d67ff9aed admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:51.103115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:51.113164 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a04371-b2ff-4d98-9103-c364b6ad5cf8 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:51.119179 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dc0aad-07a6-4d51-9432-3fdae6e4277b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:51.119824 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dc0aad-07a6-4d51-9432-3fdae6e4277b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:51.120115 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dc0aad-07a6-4d51-9432-3fdae6e4277b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:51.120633 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dc0aad-07a6-4d51-9432-3fdae6e4277b admin admin] PATH_INFO: `/v3/projects/9a686dae16a04996ad351ddb92875e0e` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:51.122685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dc0aad-07a6-4d51-9432-3fdae6e4277b admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a686dae16a04996ad351ddb92875e0e)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:51.124737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dc0aad-07a6-4d51-9432-3fdae6e4277b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:51.346908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a2dc0aad-07a6-4d51-9432-3fdae6e4277b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a686dae16a04996ad351ddb92875e0e', 'request_id': 'req-a2dc0aad-07a6-4d51-9432-3fdae6e4277b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:51.355039 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a2dc0aad-07a6-4d51-9432-3fdae6e4277b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a686dae16a04996ad351ddb92875e0e', 'request_id': 'req-a2dc0aad-07a6-4d51-9432-3fdae6e4277b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:51.365074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a2dc0aad-07a6-4d51-9432-3fdae6e4277b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a686dae16a04996ad351ddb92875e0e', 'request_id': 'req-a2dc0aad-07a6-4d51-9432-3fdae6e4277b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:51.366908 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6e4f6765-36ce-4137-b9e1-e08d67ff9aed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ad52a20c66db493983ca762d17dc7cec'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:51.393872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6e4f6765-36ce-4137-b9e1-e08d67ff9aed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ad52a20c66db493983ca762d17dc7cec'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:51.393872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6e4f6765-36ce-4137-b9e1-e08d67ff9aed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ad52a20c66db493983ca762d17dc7cec'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:51.516890 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a2dc0aad-07a6-4d51-9432-3fdae6e4277b admin admin] The token cache is being invalidate because project 9a686dae16a04996ad351ddb92875e0e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:46:51.517189 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a2dc0aad-07a6-4d51-9432-3fdae6e4277b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:51.517889 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a2dc0aad-07a6-4d51-9432-3fdae6e4277b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:51.519933 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 933/1861] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:46:51 2022] DELETE /identity/v3/projects/9a686dae16a04996ad351ddb92875e0e => generated 0 bytes in 505 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:51.524062 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dc0aad-07a6-4d51-9432-3fdae6e4277b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:51.526202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a2dc0aad-07a6-4d51-9432-3fdae6e4277b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:51.590111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6e4f6765-36ce-4137-b9e1-e08d67ff9aed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ad52a20c66db493983ca762d17dc7cec'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:51.600216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6e4f6765-36ce-4137-b9e1-e08d67ff9aed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ad52a20c66db493983ca762d17dc7cec'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:51.614193 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 929/1862] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:46:50 2022] DELETE /identity/v3/users/ad52a20c66db493983ca762d17dc7cec => generated 0 bytes in 630 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:51.664174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dc0aad-07a6-4d51-9432-3fdae6e4277b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:51.681405 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=10,keystone:SELECT=88,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:51.762974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f6f30a-4274-4b56-a6e6-baa4e318b6a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:51.763689 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f6f30a-4274-4b56-a6e6-baa4e318b6a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:51.765012 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f6f30a-4274-4b56-a6e6-baa4e318b6a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:51.765345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f6f30a-4274-4b56-a6e6-baa4e318b6a0 admin admin] PATH_INFO: `/v3/users/efdb83070001469ea1bc5670094d9160` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:51.766856 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f6f30a-4274-4b56-a6e6-baa4e318b6a0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=efdb83070001469ea1bc5670094d9160)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:51.769509 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f6f30a-4274-4b56-a6e6-baa4e318b6a0 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:52.049546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-09f6f30a-4274-4b56-a6e6-baa4e318b6a0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'efdb83070001469ea1bc5670094d9160'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:52.050338 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-09f6f30a-4274-4b56-a6e6-baa4e318b6a0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'efdb83070001469ea1bc5670094d9160'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:52.050981 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-09f6f30a-4274-4b56-a6e6-baa4e318b6a0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'efdb83070001469ea1bc5670094d9160'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:52.062303 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-09f6f30a-4274-4b56-a6e6-baa4e318b6a0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'efdb83070001469ea1bc5670094d9160'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:52.150074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-09f6f30a-4274-4b56-a6e6-baa4e318b6a0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'efdb83070001469ea1bc5670094d9160'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:52.173734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 934/1863] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:46:51 2022] DELETE /identity/v3/users/efdb83070001469ea1bc5670094d9160 => generated 0 bytes in 650 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:52.448262 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4f6765-36ce-4137-b9e1-e08d67ff9aed admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:52.449870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-6e4f6765-36ce-4137-b9e1-e08d67ff9aed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:52.463310 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4f6765-36ce-4137-b9e1-e08d67ff9aed admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:52.538191 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196db9e7-4ee9-4775-97f1-c412003be4cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:52.538802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196db9e7-4ee9-4775-97f1-c412003be4cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:52.540020 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196db9e7-4ee9-4775-97f1-c412003be4cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:52.540306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196db9e7-4ee9-4775-97f1-c412003be4cf admin admin] PATH_INFO: `/v3/projects/22ab318fbcb645bdafd7522089e44cdb` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:52.542003 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196db9e7-4ee9-4775-97f1-c412003be4cf admin admin] RBAC: Authorizing `identity:delete_project(project_id=22ab318fbcb645bdafd7522089e44cdb)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:52.543988 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196db9e7-4ee9-4775-97f1-c412003be4cf admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:52.696534 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f6f30a-4274-4b56-a6e6-baa4e318b6a0 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:52.698259 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-09f6f30a-4274-4b56-a6e6-baa4e318b6a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:52.707632 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-196db9e7-4ee9-4775-97f1-c412003be4cf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '22ab318fbcb645bdafd7522089e44cdb', 'request_id': 'req-196db9e7-4ee9-4775-97f1-c412003be4cf'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:52.713760 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-196db9e7-4ee9-4775-97f1-c412003be4cf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '22ab318fbcb645bdafd7522089e44cdb', 'request_id': 'req-196db9e7-4ee9-4775-97f1-c412003be4cf'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:52.721015 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f6f30a-4274-4b56-a6e6-baa4e318b6a0 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:52.722529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-196db9e7-4ee9-4775-97f1-c412003be4cf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '22ab318fbcb645bdafd7522089e44cdb', 'request_id': 'req-196db9e7-4ee9-4775-97f1-c412003be4cf'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:52.784199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e09de7-14b1-4b62-9955-be591968524a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:52.784199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e09de7-14b1-4b62-9955-be591968524a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:52.784199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e09de7-14b1-4b62-9955-be591968524a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:52.784199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e09de7-14b1-4b62-9955-be591968524a admin admin] PATH_INFO: `/v3/projects/7e9cca2169e84930adf5d41582bc75e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:52.784965 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e09de7-14b1-4b62-9955-be591968524a admin admin] RBAC: Authorizing `identity:delete_project(project_id=7e9cca2169e84930adf5d41582bc75e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:52.787185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e09de7-14b1-4b62-9955-be591968524a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:52.840059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-196db9e7-4ee9-4775-97f1-c412003be4cf admin admin] The token cache is being invalidate because project 22ab318fbcb645bdafd7522089e44cdb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:46:52.843783 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-196db9e7-4ee9-4775-97f1-c412003be4cf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:52.843783 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-196db9e7-4ee9-4775-97f1-c412003be4cf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:52.843783 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 930/1864] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:46:52 2022] DELETE /identity/v3/projects/22ab318fbcb645bdafd7522089e44cdb => generated 0 bytes in 395 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:52.847799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0c5e6e-7c73-4378-ab53-c71dd208717b None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:52.848088 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0c5e6e-7c73-4378-ab53-c71dd208717b None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:52.848365 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0c5e6e-7c73-4378-ab53-c71dd208717b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:52.867691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-7b0c5e6e-7c73-4378-ab53-c71dd208717b None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:52.868911 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7b0c5e6e-7c73-4378-ab53-c71dd208717b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:52.877087 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 931/1865] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:52.884592 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cec48f-2f63-42c7-95a3-35b950f4b63b None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:52.886196 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cec48f-2f63-42c7-95a3-35b950f4b63b None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:52.886196 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cec48f-2f63-42c7-95a3-35b950f4b63b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:52.893915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:52.906071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-24cec48f-2f63-42c7-95a3-35b950f4b63b None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:52.907441 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-24cec48f-2f63-42c7-95a3-35b950f4b63b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:52.921863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-88e09de7-14b1-4b62-9955-be591968524a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7e9cca2169e84930adf5d41582bc75e0', 'request_id': 'req-88e09de7-14b1-4b62-9955-be591968524a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:52.930019 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-88e09de7-14b1-4b62-9955-be591968524a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7e9cca2169e84930adf5d41582bc75e0', 'request_id': 'req-88e09de7-14b1-4b62-9955-be591968524a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:52.936809 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-88e09de7-14b1-4b62-9955-be591968524a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7e9cca2169e84930adf5d41582bc75e0', 'request_id': 'req-88e09de7-14b1-4b62-9955-be591968524a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:52.969843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 932/1866] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:52.974043 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cec48f-2f63-42c7-95a3-35b950f4b63b None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:52.983105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cec48f-2f63-42c7-95a3-35b950f4b63b None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:52.987715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1627c9-7098-4c7d-a3fa-3709a8d70fd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:52.988338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1627c9-7098-4c7d-a3fa-3709a8d70fd5 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:52.988619 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1627c9-7098-4c7d-a3fa-3709a8d70fd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:52.988892 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1627c9-7098-4c7d-a3fa-3709a8d70fd5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:52.989527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1627c9-7098-4c7d-a3fa-3709a8d70fd5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:46:52.989827 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1627c9-7098-4c7d-a3fa-3709a8d70fd5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:52.991469 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1627c9-7098-4c7d-a3fa-3709a8d70fd5 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:53.003385 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 933/1867] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:46:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:46:53.007459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1627c9-7098-4c7d-a3fa-3709a8d70fd5 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:53.017565 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1627c9-7098-4c7d-a3fa-3709a8d70fd5 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:53.022144 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1e5b8c-a106-4afc-a4f2-0a1d4f4362f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:53.022732 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1e5b8c-a106-4afc-a4f2-0a1d4f4362f6 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:53.023017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1e5b8c-a106-4afc-a4f2-0a1d4f4362f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:53.023293 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1e5b8c-a106-4afc-a4f2-0a1d4f4362f6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:53.023895 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1e5b8c-a106-4afc-a4f2-0a1d4f4362f6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:53.025737 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1e5b8c-a106-4afc-a4f2-0a1d4f4362f6 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:53.036623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-88e09de7-14b1-4b62-9955-be591968524a admin admin] The token cache is being invalidate because project 7e9cca2169e84930adf5d41582bc75e0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:46:53.040231 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-88e09de7-14b1-4b62-9955-be591968524a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:53.040231 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-88e09de7-14b1-4b62-9955-be591968524a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:53.040231 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 935/1868] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:46:52 2022] DELETE /identity/v3/projects/7e9cca2169e84930adf5d41582bc75e0 => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:53.044015 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a4c9a3-436f-47c0-9855-cd2c5c039b7b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:53.044297 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a4c9a3-436f-47c0-9855-cd2c5c039b7b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:53.044563 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a4c9a3-436f-47c0-9855-cd2c5c039b7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:53.064590 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-51a4c9a3-436f-47c0-9855-cd2c5c039b7b None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:53.068507 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-51a4c9a3-436f-47c0-9855-cd2c5c039b7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:53.081113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 936/1869] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:53.082289 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd702bb-2e44-40a7-9417-4e9daf9a041d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:53.082576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd702bb-2e44-40a7-9417-4e9daf9a041d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:53.082898 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd702bb-2e44-40a7-9417-4e9daf9a041d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:53.108173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-fcd702bb-2e44-40a7-9417-4e9daf9a041d None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:53.109543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fcd702bb-2e44-40a7-9417-4e9daf9a041d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:53.118441 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 937/1870] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:53.122558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd702bb-2e44-40a7-9417-4e9daf9a041d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:53.132887 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd702bb-2e44-40a7-9417-4e9daf9a041d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:53.139579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da1bf2f-d553-4a39-8be9-247913a1b6df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:53.140254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da1bf2f-d553-4a39-8be9-247913a1b6df admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:53.140919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da1bf2f-d553-4a39-8be9-247913a1b6df admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:53.141284 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da1bf2f-d553-4a39-8be9-247913a1b6df admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:53.143014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da1bf2f-d553-4a39-8be9-247913a1b6df admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:46:53.143014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da1bf2f-d553-4a39-8be9-247913a1b6df admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:53.145133 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da1bf2f-d553-4a39-8be9-247913a1b6df admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:53.157952 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 934/1871] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:53 2022] POST /identity/v3/projects => generated 373 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:46:53.161991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1e5b8c-a106-4afc-a4f2-0a1d4f4362f6 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:53.164941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-bc1e5b8c-a106-4afc-a4f2-0a1d4f4362f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:53.167584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 938/1872] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:46:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:46:53.173491 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da1bf2f-d553-4a39-8be9-247913a1b6df admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:53.187975 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da1bf2f-d553-4a39-8be9-247913a1b6df admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:53.253187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1e5b8c-a106-4afc-a4f2-0a1d4f4362f6 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:53.259572 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b5e390-f807-4880-90fc-c1d267bcaba0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:53.260199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b5e390-f807-4880-90fc-c1d267bcaba0 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:53.260647 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b5e390-f807-4880-90fc-c1d267bcaba0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:53.260926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b5e390-f807-4880-90fc-c1d267bcaba0 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:53.261575 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b5e390-f807-4880-90fc-c1d267bcaba0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:53.261966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a898ffd-35e6-4758-a69f-e6116e09a9c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:53.262872 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a898ffd-35e6-4758-a69f-e6116e09a9c1 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:53.267407 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a898ffd-35e6-4758-a69f-e6116e09a9c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:53.267407 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a898ffd-35e6-4758-a69f-e6116e09a9c1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:53.267662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b5e390-f807-4880-90fc-c1d267bcaba0 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:53.267757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a898ffd-35e6-4758-a69f-e6116e09a9c1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:53.267757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a898ffd-35e6-4758-a69f-e6116e09a9c1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:53.359837 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 935/1873] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:46:53 2022] POST /identity/v3/users => generated 287 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:46:53.360871 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b5e390-f807-4880-90fc-c1d267bcaba0 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:53.378950 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b5e390-f807-4880-90fc-c1d267bcaba0 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:53.392204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 939/1874] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:53 2022] POST /identity/v3/projects => generated 381 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:46:53.399557 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357f45f3-7807-41dc-8233-1090c9d97c93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:53.400348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a898ffd-35e6-4758-a69f-e6116e09a9c1 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:53.400474 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357f45f3-7807-41dc-8233-1090c9d97c93 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:53.400645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357f45f3-7807-41dc-8233-1090c9d97c93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:53.401040 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357f45f3-7807-41dc-8233-1090c9d97c93 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:53.401842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357f45f3-7807-41dc-8233-1090c9d97c93 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:53.403584 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357f45f3-7807-41dc-8233-1090c9d97c93 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:53.414096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 936/1875] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:53 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:53.418022 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357f45f3-7807-41dc-8233-1090c9d97c93 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:53.430293 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357f45f3-7807-41dc-8233-1090c9d97c93 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:53.476020 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a898ffd-35e6-4758-a69f-e6116e09a9c1 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:53.476020 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6bdef8-8102-4a80-a228-59ba491842b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:53.476020 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6bdef8-8102-4a80-a228-59ba491842b7 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:53.476020 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6bdef8-8102-4a80-a228-59ba491842b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:53.476382 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6bdef8-8102-4a80-a228-59ba491842b7 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:53.476382 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6bdef8-8102-4a80-a228-59ba491842b7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:53.478217 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6bdef8-8102-4a80-a228-59ba491842b7 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:53.511483 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e468cbf2-0d29-4240-b901-7daceb8c9c3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:53.512098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e468cbf2-0d29-4240-b901-7daceb8c9c3f admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:53.512667 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e468cbf2-0d29-4240-b901-7daceb8c9c3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:53.512948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e468cbf2-0d29-4240-b901-7daceb8c9c3f admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:53.513592 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e468cbf2-0d29-4240-b901-7daceb8c9c3f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:53.515283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e468cbf2-0d29-4240-b901-7daceb8c9c3f admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:53.526628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 937/1876] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:53 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:46:53.529694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e468cbf2-0d29-4240-b901-7daceb8c9c3f admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:53.539307 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e468cbf2-0d29-4240-b901-7daceb8c9c3f admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:53.543979 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6c92e4-53aa-4b66-b5ca-820ceb2b7c7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:53.544617 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6c92e4-53aa-4b66-b5ca-820ceb2b7c7a admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:53.544899 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6c92e4-53aa-4b66-b5ca-820ceb2b7c7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:53.545701 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6c92e4-53aa-4b66-b5ca-820ceb2b7c7a admin admin] PATH_INFO: `/v3/projects/be14f805af7a4180aea48756132410a5/users/353e508f25e14434954de3ee4670f135/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:53.550338 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:53.560704 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6c92e4-53aa-4b66-b5ca-820ceb2b7c7a admin admin] RBAC: Authorizing `identity:create_grant(project_id=be14f805af7a4180aea48756132410a5, user_id=353e508f25e14434954de3ee4670f135, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:53.564127 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6c92e4-53aa-4b66-b5ca-820ceb2b7c7a admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:53.586627 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 940/1877] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:46:53 2022] POST /identity/v3/users => generated 291 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:53.589394 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6bdef8-8102-4a80-a228-59ba491842b7 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:53.600096 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6bdef8-8102-4a80-a228-59ba491842b7 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:53.604675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7d759e-9e29-41e0-b2f5-7a2b832c2120 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:53.605275 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7d759e-9e29-41e0-b2f5-7a2b832c2120 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:53.605552 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7d759e-9e29-41e0-b2f5-7a2b832c2120 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:53.605936 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7d759e-9e29-41e0-b2f5-7a2b832c2120 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:53.606531 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7d759e-9e29-41e0-b2f5-7a2b832c2120 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:53.608680 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7d759e-9e29-41e0-b2f5-7a2b832c2120 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:53.619036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 941/1878] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:53 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:46:53.623818 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7d759e-9e29-41e0-b2f5-7a2b832c2120 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:53.633561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7d759e-9e29-41e0-b2f5-7a2b832c2120 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:53.639809 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9e9a32-0a52-4102-a256-e04f686853ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:53.640668 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9e9a32-0a52-4102-a256-e04f686853ef admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:53.640943 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9e9a32-0a52-4102-a256-e04f686853ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:53.641608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9e9a32-0a52-4102-a256-e04f686853ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:53.642253 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9e9a32-0a52-4102-a256-e04f686853ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:53.643914 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9e9a32-0a52-4102-a256-e04f686853ef admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:53.655462 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 942/1879] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:53 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:53.659570 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9e9a32-0a52-4102-a256-e04f686853ef admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:53.674847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9e9a32-0a52-4102-a256-e04f686853ef admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:53.674951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 938/1880] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:53 2022] PUT /identity/v3/projects/be14f805af7a4180aea48756132410a5/users/353e508f25e14434954de3ee4670f135/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:53.685653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1357b4ba-0432-4639-9ec1-7b5d9a98b5e1 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:53.687089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1357b4ba-0432-4639-9ec1-7b5d9a98b5e1 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:53.687386 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1357b4ba-0432-4639-9ec1-7b5d9a98b5e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:53.724133 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-1357b4ba-0432-4639-9ec1-7b5d9a98b5e1 None None] MFA Rules not processed for user `353e508f25e14434954de3ee4670f135`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:53.727339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-1357b4ba-0432-4639-9ec1-7b5d9a98b5e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:53.763066 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2177e755-7b22-466d-87d4-8f2e298ebf49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:53.764062 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2177e755-7b22-466d-87d4-8f2e298ebf49 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:53.764490 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2177e755-7b22-466d-87d4-8f2e298ebf49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:53.764774 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2177e755-7b22-466d-87d4-8f2e298ebf49 admin admin] PATH_INFO: `/v3/projects/02501f755ba54c339a3b8808b8f0820e/users/d13271da32404e2d8d706e2e56bdb054/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:53.799813 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2177e755-7b22-466d-87d4-8f2e298ebf49 admin admin] RBAC: Authorizing `identity:create_grant(project_id=02501f755ba54c339a3b8808b8f0820e, user_id=d13271da32404e2d8d706e2e56bdb054, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:53.803423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2177e755-7b22-466d-87d4-8f2e298ebf49 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:53.823152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 939/1881] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:53 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:53.825012 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1357b4ba-0432-4639-9ec1-7b5d9a98b5e1 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:53.837786 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1357b4ba-0432-4639-9ec1-7b5d9a98b5e1 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:53.843320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c51e17-5a50-4eb6-aec0-270df8d46ea7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:53.843941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c51e17-5a50-4eb6-aec0-270df8d46ea7 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:53.844226 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c51e17-5a50-4eb6-aec0-270df8d46ea7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:53.844502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c51e17-5a50-4eb6-aec0-270df8d46ea7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:53.845142 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c51e17-5a50-4eb6-aec0-270df8d46ea7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:53.847417 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c51e17-5a50-4eb6-aec0-270df8d46ea7 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:53.922524 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 943/1882] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:53 2022] PUT /identity/v3/projects/02501f755ba54c339a3b8808b8f0820e/users/d13271da32404e2d8d706e2e56bdb054/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:53.932440 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f9a272-49a3-4a14-b052-08a4389483da None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:53.932722 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f9a272-49a3-4a14-b052-08a4389483da None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:53.932996 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f9a272-49a3-4a14-b052-08a4389483da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:53.961128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-89f9a272-49a3-4a14-b052-08a4389483da None None] MFA Rules not processed for user `d13271da32404e2d8d706e2e56bdb054`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:53.962388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-89f9a272-49a3-4a14-b052-08a4389483da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:53.962888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 940/1883] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:53 2022] POST /identity/v3/projects => generated 373 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:53.966683 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c51e17-5a50-4eb6-aec0-270df8d46ea7 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:54.033459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c51e17-5a50-4eb6-aec0-270df8d46ea7 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:54.038142 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 944/1884] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:53 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:54.038355 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d8a549-fdaf-4ea0-88b1-25c141e4abac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:54.039117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d8a549-fdaf-4ea0-88b1-25c141e4abac admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:54.039424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d8a549-fdaf-4ea0-88b1-25c141e4abac admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:54.039752 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d8a549-fdaf-4ea0-88b1-25c141e4abac admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:54.040381 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d8a549-fdaf-4ea0-88b1-25c141e4abac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:54.042327 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d8a549-fdaf-4ea0-88b1-25c141e4abac admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:54.042765 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f9a272-49a3-4a14-b052-08a4389483da None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:54.053859 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f9a272-49a3-4a14-b052-08a4389483da None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:54.058397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dbf2d5-d488-4b48-9d23-5061e766cd48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:54.059074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dbf2d5-d488-4b48-9d23-5061e766cd48 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:54.059349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dbf2d5-d488-4b48-9d23-5061e766cd48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:54.059621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dbf2d5-d488-4b48-9d23-5061e766cd48 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:54.060242 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7dbf2d5-d488-4b48-9d23-5061e766cd48 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:54.062154 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7dbf2d5-d488-4b48-9d23-5061e766cd48 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:54.133482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 941/1885] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:46:53 2022] POST /identity/v3/users => generated 293 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:54.137505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d8a549-fdaf-4ea0-88b1-25c141e4abac admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:54.147462 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d8a549-fdaf-4ea0-88b1-25c141e4abac admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:54.152055 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56200dda-cf81-49ad-b21c-9d903e31b8b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:54.152624 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56200dda-cf81-49ad-b21c-9d903e31b8b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:54.152942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56200dda-cf81-49ad-b21c-9d903e31b8b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:54.153219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56200dda-cf81-49ad-b21c-9d903e31b8b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:54.153843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56200dda-cf81-49ad-b21c-9d903e31b8b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:54.156083 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56200dda-cf81-49ad-b21c-9d903e31b8b6 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:54.165717 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 942/1886] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:54 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:46:54.168580 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 945/1887] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:54 2022] POST /identity/v3/projects => generated 379 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:46:54.169815 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56200dda-cf81-49ad-b21c-9d903e31b8b6 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:54.172367 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dbf2d5-d488-4b48-9d23-5061e766cd48 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:54.179654 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56200dda-cf81-49ad-b21c-9d903e31b8b6 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:54.248083 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dbf2d5-d488-4b48-9d23-5061e766cd48 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:54.253417 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec07945-f5ff-42f1-b328-c919edda908d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:54.258454 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec07945-f5ff-42f1-b328-c919edda908d admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:54.258833 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec07945-f5ff-42f1-b328-c919edda908d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:54.259121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec07945-f5ff-42f1-b328-c919edda908d admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:54.259905 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec07945-f5ff-42f1-b328-c919edda908d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:54.261868 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec07945-f5ff-42f1-b328-c919edda908d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:54.285209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8463a5e8-4639-48cc-b15f-63af2ef7f585 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:54.285890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8463a5e8-4639-48cc-b15f-63af2ef7f585 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:54.286170 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8463a5e8-4639-48cc-b15f-63af2ef7f585 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:54.286452 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8463a5e8-4639-48cc-b15f-63af2ef7f585 admin admin] PATH_INFO: `/v3/projects/a0b1b0614863438c99c08917ec713768/users/ec2bd6ae792143c094f7657c6d6ec745/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:54.301637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8463a5e8-4639-48cc-b15f-63af2ef7f585 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0b1b0614863438c99c08917ec713768, user_id=ec2bd6ae792143c094f7657c6d6ec745, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:54.304402 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8463a5e8-4639-48cc-b15f-63af2ef7f585 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:54.356410 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 946/1888] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:46:54 2022] POST /identity/v3/users => generated 296 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:46:54.360164 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec07945-f5ff-42f1-b328-c919edda908d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:54.369960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec07945-f5ff-42f1-b328-c919edda908d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:54.374894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412eac32-d407-4849-a793-5c5d63de24e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:54.375455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412eac32-d407-4849-a793-5c5d63de24e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:54.375730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412eac32-d407-4849-a793-5c5d63de24e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:54.376002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412eac32-d407-4849-a793-5c5d63de24e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:54.376618 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412eac32-d407-4849-a793-5c5d63de24e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:54.378263 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412eac32-d407-4849-a793-5c5d63de24e3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:54.389411 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 947/1889] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:54 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:46:54.392102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412eac32-d407-4849-a793-5c5d63de24e3 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:54.401336 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412eac32-d407-4849-a793-5c5d63de24e3 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:54.406797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badda08f-7da6-4082-b567-e8c60496bbb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:54.407709 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badda08f-7da6-4082-b567-e8c60496bbb1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:54.407998 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badda08f-7da6-4082-b567-e8c60496bbb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:54.408268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badda08f-7da6-4082-b567-e8c60496bbb1 admin admin] PATH_INFO: `/v3/projects/f3ce98276abc499fbd2a329e523fa124/users/46b1d4c8ad2e41d194789555f09727a5/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:54.425508 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 943/1890] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:54 2022] PUT /identity/v3/projects/a0b1b0614863438c99c08917ec713768/users/ec2bd6ae792143c094f7657c6d6ec745/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:54.428551 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badda08f-7da6-4082-b567-e8c60496bbb1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f3ce98276abc499fbd2a329e523fa124, user_id=46b1d4c8ad2e41d194789555f09727a5, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:54.429509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8463a5e8-4639-48cc-b15f-63af2ef7f585 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:54.431326 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badda08f-7da6-4082-b567-e8c60496bbb1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:54.496047 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8463a5e8-4639-48cc-b15f-63af2ef7f585 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:54.501447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ae517c-b319-46c1-83c6-c21cbf9d881d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:54.502056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ae517c-b319-46c1-83c6-c21cbf9d881d admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:54.502332 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ae517c-b319-46c1-83c6-c21cbf9d881d admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:54.502626 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ae517c-b319-46c1-83c6-c21cbf9d881d admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:54.503280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ae517c-b319-46c1-83c6-c21cbf9d881d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:54.504932 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ae517c-b319-46c1-83c6-c21cbf9d881d admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:54.516953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 944/1891] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:54 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:54.517791 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ae517c-b319-46c1-83c6-c21cbf9d881d admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:54.520027 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-96ae517c-b319-46c1-83c6-c21cbf9d881d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:54.527859 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 948/1892] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:54 2022] PUT /identity/v3/projects/f3ce98276abc499fbd2a329e523fa124/users/46b1d4c8ad2e41d194789555f09727a5/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:54.534664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ae517c-b319-46c1-83c6-c21cbf9d881d admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:54.535239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badda08f-7da6-4082-b567-e8c60496bbb1 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:54.596419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5195508f-addb-4506-a665-a985ae201ec8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:54.597053 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5195508f-addb-4506-a665-a985ae201ec8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:54.597649 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5195508f-addb-4506-a665-a985ae201ec8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:54.597649 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5195508f-addb-4506-a665-a985ae201ec8 admin admin] PATH_INFO: `/v3/users/9840276541c04f1abcf2e679da45ec33` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:54.598725 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5195508f-addb-4506-a665-a985ae201ec8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9840276541c04f1abcf2e679da45ec33)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:54.600648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5195508f-addb-4506-a665-a985ae201ec8 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:54.621237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badda08f-7da6-4082-b567-e8c60496bbb1 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:54.626576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0b7054-a82a-4d50-89af-8073160b3539 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:54.627225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0b7054-a82a-4d50-89af-8073160b3539 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:54.627503 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0b7054-a82a-4d50-89af-8073160b3539 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:54.627769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0b7054-a82a-4d50-89af-8073160b3539 admin admin] PATH_INFO: `/v3/projects/a0b1b0614863438c99c08917ec713768/users/ec2bd6ae792143c094f7657c6d6ec745/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:54.629677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0b7054-a82a-4d50-89af-8073160b3539 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0b1b0614863438c99c08917ec713768, user_id=ec2bd6ae792143c094f7657c6d6ec745, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:54.632392 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0b7054-a82a-4d50-89af-8073160b3539 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:54.643219 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 949/1893] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:54 2022] PUT /identity/v3/projects/a0b1b0614863438c99c08917ec713768/users/ec2bd6ae792143c094f7657c6d6ec745/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:54.644265 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0b7054-a82a-4d50-89af-8073160b3539 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:54.704556 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0b7054-a82a-4d50-89af-8073160b3539 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:54.709056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b142ea85-21f6-4e21-a4b5-f569877f1005 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:54.709649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b142ea85-21f6-4e21-a4b5-f569877f1005 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:54.709924 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b142ea85-21f6-4e21-a4b5-f569877f1005 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:54.710194 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b142ea85-21f6-4e21-a4b5-f569877f1005 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:54.711122 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b142ea85-21f6-4e21-a4b5-f569877f1005 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:54.712702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b142ea85-21f6-4e21-a4b5-f569877f1005 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:54.722155 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 950/1894] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:54 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:54.723465 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b3a4b2-8652-43ea-b493-a185e70f66c2 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:54.723826 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b3a4b2-8652-43ea-b493-a185e70f66c2 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:54.724102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b3a4b2-8652-43ea-b493-a185e70f66c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:54.752106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-16b3a4b2-8652-43ea-b493-a185e70f66c2 None None] MFA Rules not processed for user `ec2bd6ae792143c094f7657c6d6ec745`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:54.753307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-16b3a4b2-8652-43ea-b493-a185e70f66c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:54.759914 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:54.819123 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 951/1895] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:54 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:54.820066 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b3a4b2-8652-43ea-b493-a185e70f66c2 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:54.833496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b3a4b2-8652-43ea-b493-a185e70f66c2 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:54.840265 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba2ddef-9b9f-408f-9765-3763ce65e801 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:54.841060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba2ddef-9b9f-408f-9765-3763ce65e801 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:54.841233 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5195508f-addb-4506-a665-a985ae201ec8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9840276541c04f1abcf2e679da45ec33'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:54.841526 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba2ddef-9b9f-408f-9765-3763ce65e801 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:54.841933 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba2ddef-9b9f-408f-9765-3763ce65e801 admin admin] PATH_INFO: `/v3/projects/f3ce98276abc499fbd2a329e523fa124/users/46b1d4c8ad2e41d194789555f09727a5/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:54.844537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba2ddef-9b9f-408f-9765-3763ce65e801 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f3ce98276abc499fbd2a329e523fa124, user_id=46b1d4c8ad2e41d194789555f09727a5, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:54.847253 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba2ddef-9b9f-408f-9765-3763ce65e801 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:54.853177 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5195508f-addb-4506-a665-a985ae201ec8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9840276541c04f1abcf2e679da45ec33'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:54.853831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5195508f-addb-4506-a665-a985ae201ec8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9840276541c04f1abcf2e679da45ec33'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:54.857958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 952/1896] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:54 2022] PUT /identity/v3/projects/f3ce98276abc499fbd2a329e523fa124/users/46b1d4c8ad2e41d194789555f09727a5/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:54.868010 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba2ddef-9b9f-408f-9765-3763ce65e801 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:54.868010 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fba2ddef-9b9f-408f-9765-3763ce65e801 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:54.919504 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5195508f-addb-4506-a665-a985ae201ec8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9840276541c04f1abcf2e679da45ec33'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:54.923361 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba2ddef-9b9f-408f-9765-3763ce65e801 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:54.925618 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee967320-93b9-4bd3-8bde-f8fab8f2e0f2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:54.926218 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee967320-93b9-4bd3-8bde-f8fab8f2e0f2 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:54.926498 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee967320-93b9-4bd3-8bde-f8fab8f2e0f2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:54.926795 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee967320-93b9-4bd3-8bde-f8fab8f2e0f2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:54.928612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ee967320-93b9-4bd3-8bde-f8fab8f2e0f2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:54.929566 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5195508f-addb-4506-a665-a985ae201ec8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9840276541c04f1abcf2e679da45ec33'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:54.936492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 945/1897] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:46:54 2022] DELETE /identity/v3/users/9840276541c04f1abcf2e679da45ec33 => generated 0 bytes in 419 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:54.939497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ea4001-7e85-48f6-9b1d-9fcd481daee6 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:54.939497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ea4001-7e85-48f6-9b1d-9fcd481daee6 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:54.939497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ea4001-7e85-48f6-9b1d-9fcd481daee6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:54.970095 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-11ea4001-7e85-48f6-9b1d-9fcd481daee6 None None] MFA Rules not processed for user `46b1d4c8ad2e41d194789555f09727a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:54.972465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-11ea4001-7e85-48f6-9b1d-9fcd481daee6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:54.995519 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:55.012051 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee967320-93b9-4bd3-8bde-f8fab8f2e0f2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:55.013855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee967320-93b9-4bd3-8bde-f8fab8f2e0f2 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:55.024217 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 953/1898] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:54 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:46:55.026547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cc4d76-0647-4369-b1a0-17decd01c1c0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:55.026885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cc4d76-0647-4369-b1a0-17decd01c1c0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:55.027172 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cc4d76-0647-4369-b1a0-17decd01c1c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:55.052188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 946/1899] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:54 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:55.058131 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-81cc4d76-0647-4369-b1a0-17decd01c1c0 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:55.059442 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-81cc4d76-0647-4369-b1a0-17decd01c1c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:55.066305 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ea4001-7e85-48f6-9b1d-9fcd481daee6 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:55.067830 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-11ea4001-7e85-48f6-9b1d-9fcd481daee6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:55.139257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ea4001-7e85-48f6-9b1d-9fcd481daee6 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:55.144051 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e29151-c2d4-40fc-a7fa-0990aeaff035 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:55.144662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e29151-c2d4-40fc-a7fa-0990aeaff035 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:55.144943 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e29151-c2d4-40fc-a7fa-0990aeaff035 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:55.145312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 954/1900] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:55.148333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e29151-c2d4-40fc-a7fa-0990aeaff035 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:55.154886 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81cc4d76-0647-4369-b1a0-17decd01c1c0 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:55.156387 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-81cc4d76-0647-4369-b1a0-17decd01c1c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:55.158935 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e29151-c2d4-40fc-a7fa-0990aeaff035 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:55.160751 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e29151-c2d4-40fc-a7fa-0990aeaff035 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:55.173624 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:55.182624 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 947/1901] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:55 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:46:55.232878 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81cc4d76-0647-4369-b1a0-17decd01c1c0 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:55.238276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2983273f-ad95-4ba9-bfec-9ab4fd772180 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:55.238276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2983273f-ad95-4ba9-bfec-9ab4fd772180 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:55.238276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2983273f-ad95-4ba9-bfec-9ab4fd772180 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:55.238276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2983273f-ad95-4ba9-bfec-9ab4fd772180 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:55.247751 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2983273f-ad95-4ba9-bfec-9ab4fd772180 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:55.249762 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2983273f-ad95-4ba9-bfec-9ab4fd772180 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:55.259607 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 955/1902] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:46:55.712254 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e29151-c2d4-40fc-a7fa-0990aeaff035 service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:55.713793 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-41e29151-c2d4-40fc-a7fa-0990aeaff035 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:55.726661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:55.750124 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e29151-c2d4-40fc-a7fa-0990aeaff035 service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:55.755458 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9752a9b-bc4a-4007-8075-c77b4db765a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:55.755667 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9752a9b-bc4a-4007-8075-c77b4db765a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:55.762175 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9752a9b-bc4a-4007-8075-c77b4db765a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:55.762175 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9752a9b-bc4a-4007-8075-c77b4db765a2 admin admin] PATH_INFO: `/v3/projects/179cdac0ebe64e4ab4c01578c80c7845` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:55.762175 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9752a9b-bc4a-4007-8075-c77b4db765a2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=179cdac0ebe64e4ab4c01578c80c7845)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:55.762175 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9752a9b-bc4a-4007-8075-c77b4db765a2 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:55.807508 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=94,keystone:INSERT=1,keystone:DELETE=5 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:55.883176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e9752a9b-bc4a-4007-8075-c77b4db765a2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '179cdac0ebe64e4ab4c01578c80c7845', 'request_id': 'req-e9752a9b-bc4a-4007-8075-c77b4db765a2'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:55.889246 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e9752a9b-bc4a-4007-8075-c77b4db765a2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '179cdac0ebe64e4ab4c01578c80c7845', 'request_id': 'req-e9752a9b-bc4a-4007-8075-c77b4db765a2'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:55.895632 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e9752a9b-bc4a-4007-8075-c77b4db765a2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '179cdac0ebe64e4ab4c01578c80c7845', 'request_id': 'req-e9752a9b-bc4a-4007-8075-c77b4db765a2'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:55.986282 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e9752a9b-bc4a-4007-8075-c77b4db765a2 admin admin] The token cache is being invalidate because project 179cdac0ebe64e4ab4c01578c80c7845 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:46:55.986583 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e9752a9b-bc4a-4007-8075-c77b4db765a2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:55.987267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e9752a9b-bc4a-4007-8075-c77b4db765a2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:55.991693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 948/1903] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:46:55 2022] DELETE /identity/v3/projects/179cdac0ebe64e4ab4c01578c80c7845 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:55.993693 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2983273f-ad95-4ba9-bfec-9ab4fd772180 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:56.005329 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2983273f-ad95-4ba9-bfec-9ab4fd772180 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:56.066204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8795546-3c7b-47f9-b954-be8689d68ac5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:56.066867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8795546-3c7b-47f9-b954-be8689d68ac5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:56.067145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8795546-3c7b-47f9-b954-be8689d68ac5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:56.067414 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8795546-3c7b-47f9-b954-be8689d68ac5 admin admin] PATH_INFO: `/v3/users/cd9ae696a0484c558fe101bc98682914` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:56.068565 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8795546-3c7b-47f9-b954-be8689d68ac5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cd9ae696a0484c558fe101bc98682914)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:56.070467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8795546-3c7b-47f9-b954-be8689d68ac5 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:56.278846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b8795546-3c7b-47f9-b954-be8689d68ac5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cd9ae696a0484c558fe101bc98682914'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:56.279501 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b8795546-3c7b-47f9-b954-be8689d68ac5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cd9ae696a0484c558fe101bc98682914'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:56.280152 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b8795546-3c7b-47f9-b954-be8689d68ac5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cd9ae696a0484c558fe101bc98682914'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:56.287149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b8795546-3c7b-47f9-b954-be8689d68ac5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cd9ae696a0484c558fe101bc98682914'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:56.349999 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b8795546-3c7b-47f9-b954-be8689d68ac5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cd9ae696a0484c558fe101bc98682914'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:56.366850 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 956/1904] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:46:55 2022] DELETE /identity/v3/users/cd9ae696a0484c558fe101bc98682914 => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:56.379908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:56.734006 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:56.742997 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9752a9b-bc4a-4007-8075-c77b4db765a2 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:56.744511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-e9752a9b-bc4a-4007-8075-c77b4db765a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:56.817433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9752a9b-bc4a-4007-8075-c77b4db765a2 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:56.873593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8795546-3c7b-47f9-b954-be8689d68ac5 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:56.875915 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b8795546-3c7b-47f9-b954-be8689d68ac5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:56.890699 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082a6e7b-30fb-4fa2-85fd-aa487ca49a57 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '46b1d4c8ad2e41d194789555f09727a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f3ce98276abc499fbd2a329e523fa124', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:56.891332 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082a6e7b-30fb-4fa2-85fd-aa487ca49a57 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:56.891711 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082a6e7b-30fb-4fa2-85fd-aa487ca49a57 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:56.891997 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082a6e7b-30fb-4fa2-85fd-aa487ca49a57 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:56.892798 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082a6e7b-30fb-4fa2-85fd-aa487ca49a57 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:46:56.893111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082a6e7b-30fb-4fa2-85fd-aa487ca49a57 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:56.894804 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082a6e7b-30fb-4fa2-85fd-aa487ca49a57 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:56.919090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 949/1905] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:46:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:46:56.920844 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082a6e7b-30fb-4fa2-85fd-aa487ca49a57 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:56.922620 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-082a6e7b-30fb-4fa2-85fd-aa487ca49a57 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:56.928743 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:56.941703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082a6e7b-30fb-4fa2-85fd-aa487ca49a57 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:56.950863 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286ed291-033f-4eeb-a0e1-0f2a7afda805 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '46b1d4c8ad2e41d194789555f09727a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f3ce98276abc499fbd2a329e523fa124', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:56.953010 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286ed291-033f-4eeb-a0e1-0f2a7afda805 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:56.953296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286ed291-033f-4eeb-a0e1-0f2a7afda805 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:56.953578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286ed291-033f-4eeb-a0e1-0f2a7afda805 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:56.954262 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286ed291-033f-4eeb-a0e1-0f2a7afda805 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:56.961146 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286ed291-033f-4eeb-a0e1-0f2a7afda805 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:57.004491 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8795546-3c7b-47f9-b954-be8689d68ac5 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:57.009089 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91216923-92eb-414b-8d30-ab78c3b92688 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:57.009719 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91216923-92eb-414b-8d30-ab78c3b92688 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:57.009996 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91216923-92eb-414b-8d30-ab78c3b92688 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:57.010267 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91216923-92eb-414b-8d30-ab78c3b92688 admin admin] PATH_INFO: `/v3/projects/6c2541f2d9b84a7e8b07e26eab937f72` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:57.011406 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91216923-92eb-414b-8d30-ab78c3b92688 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6c2541f2d9b84a7e8b07e26eab937f72)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:57.013325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91216923-92eb-414b-8d30-ab78c3b92688 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:57.046001 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 950/1906] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:56 2022] POST /identity/v3/projects => generated 354 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:57.049887 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286ed291-033f-4eeb-a0e1-0f2a7afda805 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:57.116370 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286ed291-033f-4eeb-a0e1-0f2a7afda805 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:57.128801 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-91216923-92eb-414b-8d30-ab78c3b92688 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6c2541f2d9b84a7e8b07e26eab937f72', 'request_id': 'req-91216923-92eb-414b-8d30-ab78c3b92688'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:57.134975 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-91216923-92eb-414b-8d30-ab78c3b92688 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6c2541f2d9b84a7e8b07e26eab937f72', 'request_id': 'req-91216923-92eb-414b-8d30-ab78c3b92688'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:57.142513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-91216923-92eb-414b-8d30-ab78c3b92688 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6c2541f2d9b84a7e8b07e26eab937f72', 'request_id': 'req-91216923-92eb-414b-8d30-ab78c3b92688'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:57.172704 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee72fcd-dbaf-48ab-929d-4cb8b61c40ba tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '46b1d4c8ad2e41d194789555f09727a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f3ce98276abc499fbd2a329e523fa124', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:57.173312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee72fcd-dbaf-48ab-929d-4cb8b61c40ba tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:57.174753 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee72fcd-dbaf-48ab-929d-4cb8b61c40ba tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:57.175077 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee72fcd-dbaf-48ab-929d-4cb8b61c40ba tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:57.175857 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee72fcd-dbaf-48ab-929d-4cb8b61c40ba tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:46:57.176173 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee72fcd-dbaf-48ab-929d-4cb8b61c40ba tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:57.177842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee72fcd-dbaf-48ab-929d-4cb8b61c40ba tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:57.189353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 951/1907] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:46:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:46:57.232305 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-91216923-92eb-414b-8d30-ab78c3b92688 admin admin] The token cache is being invalidate because project 6c2541f2d9b84a7e8b07e26eab937f72 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:46:57.232305 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-91216923-92eb-414b-8d30-ab78c3b92688 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:57.232305 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-91216923-92eb-414b-8d30-ab78c3b92688 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:57.232305 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 957/1908] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:46:56 2022] DELETE /identity/v3/projects/6c2541f2d9b84a7e8b07e26eab937f72 => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:57.232920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ceeeb7-cc32-4490-b567-bbaf52717a27 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:57.233001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91216923-92eb-414b-8d30-ab78c3b92688 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:57.233073 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ceeeb7-cc32-4490-b567-bbaf52717a27 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:57.233073 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ceeeb7-cc32-4490-b567-bbaf52717a27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:57.233264 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-91216923-92eb-414b-8d30-ab78c3b92688 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:57.246585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91216923-92eb-414b-8d30-ab78c3b92688 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:57.251365 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5742da77-c6d6-46f0-a170-8d95cd28a512 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '46b1d4c8ad2e41d194789555f09727a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f3ce98276abc499fbd2a329e523fa124', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:57.251996 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5742da77-c6d6-46f0-a170-8d95cd28a512 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:57.252278 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5742da77-c6d6-46f0-a170-8d95cd28a512 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:57.252552 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5742da77-c6d6-46f0-a170-8d95cd28a512 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] PATH_INFO: `/v3/projects/6203467f61f34ef5a4cc34c4c0a1d4ec` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:57.255178 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-07ceeeb7-cc32-4490-b567-bbaf52717a27 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:57.255292 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5742da77-c6d6-46f0-a170-8d95cd28a512 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] RBAC: Authorizing `identity:delete_project(project_id=6203467f61f34ef5a4cc34c4c0a1d4ec)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:57.255425 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-07ceeeb7-cc32-4490-b567-bbaf52717a27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:57.255608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5742da77-c6d6-46f0-a170-8d95cd28a512 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:57.316927 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 952/1909] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:57.324388 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf48f8c6-3d30-4e9b-9785-c59a1b3ab401 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:57.324703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf48f8c6-3d30-4e9b-9785-c59a1b3ab401 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:57.324956 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf48f8c6-3d30-4e9b-9785-c59a1b3ab401 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:57.347534 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-bf48f8c6-3d30-4e9b-9785-c59a1b3ab401 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:57.348943 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-bf48f8c6-3d30-4e9b-9785-c59a1b3ab401 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:57.356878 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 953/1910] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:57.368995 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf48f8c6-3d30-4e9b-9785-c59a1b3ab401 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:57.381338 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5742da77-c6d6-46f0-a170-8d95cd28a512 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6203467f61f34ef5a4cc34c4c0a1d4ec', 'request_id': 'req-5742da77-c6d6-46f0-a170-8d95cd28a512'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:57.386509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf48f8c6-3d30-4e9b-9785-c59a1b3ab401 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:57.388220 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5742da77-c6d6-46f0-a170-8d95cd28a512 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6203467f61f34ef5a4cc34c4c0a1d4ec', 'request_id': 'req-5742da77-c6d6-46f0-a170-8d95cd28a512'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:57.398187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5742da77-c6d6-46f0-a170-8d95cd28a512 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6203467f61f34ef5a4cc34c4c0a1d4ec', 'request_id': 'req-5742da77-c6d6-46f0-a170-8d95cd28a512'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:57.454367 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9a6003-1dd0-40a8-8bf7-3fe2444974f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:57.455026 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9a6003-1dd0-40a8-8bf7-3fe2444974f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:57.455301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9a6003-1dd0-40a8-8bf7-3fe2444974f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:57.455593 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9a6003-1dd0-40a8-8bf7-3fe2444974f6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:57.460135 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9a6003-1dd0-40a8-8bf7-3fe2444974f6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:46:57.460135 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9a6003-1dd0-40a8-8bf7-3fe2444974f6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:57.460135 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9a6003-1dd0-40a8-8bf7-3fe2444974f6 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:57.471771 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 954/1911] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:46:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:46:57.475839 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9a6003-1dd0-40a8-8bf7-3fe2444974f6 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:57.489012 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9a6003-1dd0-40a8-8bf7-3fe2444974f6 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:57.489192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5742da77-c6d6-46f0-a170-8d95cd28a512 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] The token cache is being invalidate because project 6203467f61f34ef5a4cc34c4c0a1d4ec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:46:57.489192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5742da77-c6d6-46f0-a170-8d95cd28a512 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:57.489192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5742da77-c6d6-46f0-a170-8d95cd28a512 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:57.489920 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 958/1912] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:46:57 2022] DELETE /identity/v3/projects/6203467f61f34ef5a4cc34c4c0a1d4ec => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:57.492131 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d4f823-9e6c-4ee8-b53f-47904b75ad5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:57.492816 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d4f823-9e6c-4ee8-b53f-47904b75ad5c admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:57.493390 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d4f823-9e6c-4ee8-b53f-47904b75ad5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:57.493542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d4f823-9e6c-4ee8-b53f-47904b75ad5c admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:57.494287 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d4f823-9e6c-4ee8-b53f-47904b75ad5c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:57.496894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d4f823-9e6c-4ee8-b53f-47904b75ad5c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:57.548300 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:57.632391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 955/1913] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:57 2022] POST /identity/v3/projects => generated 381 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:46:57.634447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5742da77-c6d6-46f0-a170-8d95cd28a512 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:57.635900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5742da77-c6d6-46f0-a170-8d95cd28a512 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:57.650096 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5742da77-c6d6-46f0-a170-8d95cd28a512 tempest-VolumeQuotaClassesTest-797787080 tempest-VolumeQuotaClassesTest-797787080-project-admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:57.710681 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2921ee9d-6885-4617-af98-c978b724641f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:57.711296 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2921ee9d-6885-4617-af98-c978b724641f admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:57.711609 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2921ee9d-6885-4617-af98-c978b724641f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:57.711912 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2921ee9d-6885-4617-af98-c978b724641f admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:57.712662 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2921ee9d-6885-4617-af98-c978b724641f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:57.714693 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2921ee9d-6885-4617-af98-c978b724641f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:57.733479 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:57.784482 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 959/1914] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:46:57 2022] POST /identity/v3/users => generated 291 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:46:57.788452 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d4f823-9e6c-4ee8-b53f-47904b75ad5c admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:57.798717 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d4f823-9e6c-4ee8-b53f-47904b75ad5c admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:57.803172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfe6bbc-465c-4cee-b4b7-aaa176e6646e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:57.803762 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfe6bbc-465c-4cee-b4b7-aaa176e6646e admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:57.804037 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfe6bbc-465c-4cee-b4b7-aaa176e6646e admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:57.804307 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfe6bbc-465c-4cee-b4b7-aaa176e6646e admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:57.804944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfe6bbc-465c-4cee-b4b7-aaa176e6646e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:57.807062 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfe6bbc-465c-4cee-b4b7-aaa176e6646e admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:57.820585 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 956/1915] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:57.821744 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2921ee9d-6885-4617-af98-c978b724641f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:57.832259 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2921ee9d-6885-4617-af98-c978b724641f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:57.838254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b9eae0-ca22-41df-ab00-1272df5ec98a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:57.838799 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b9eae0-ca22-41df-ab00-1272df5ec98a admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:57.839075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b9eae0-ca22-41df-ab00-1272df5ec98a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:57.839342 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b9eae0-ca22-41df-ab00-1272df5ec98a admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:57.840017 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b9eae0-ca22-41df-ab00-1272df5ec98a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:57.841904 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b9eae0-ca22-41df-ab00-1272df5ec98a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:57.855788 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 960/1916] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:46:57.862064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfe6bbc-465c-4cee-b4b7-aaa176e6646e admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:57.872697 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfe6bbc-465c-4cee-b4b7-aaa176e6646e admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:57.878272 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0778bd12-8c6a-4642-9a9b-7bc7c0f5edb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:57.878957 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0778bd12-8c6a-4642-9a9b-7bc7c0f5edb4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:57.879234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0778bd12-8c6a-4642-9a9b-7bc7c0f5edb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:57.879539 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0778bd12-8c6a-4642-9a9b-7bc7c0f5edb4 admin admin] PATH_INFO: `/v3/projects/3ab149fcc9c3411f8b3e08ebe2907109/users/3e526053edbf4daca2f79303999377b1/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:57.895688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0778bd12-8c6a-4642-9a9b-7bc7c0f5edb4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3ab149fcc9c3411f8b3e08ebe2907109, user_id=3e526053edbf4daca2f79303999377b1, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:57.898322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0778bd12-8c6a-4642-9a9b-7bc7c0f5edb4 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:58.007751 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 957/1917] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:57 2022] PUT /identity/v3/projects/3ab149fcc9c3411f8b3e08ebe2907109/users/3e526053edbf4daca2f79303999377b1/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:58.018212 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0370e9d9-7599-446e-8c96-2043fbd46a02 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:58.018659 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0370e9d9-7599-446e-8c96-2043fbd46a02 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:58.018922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0370e9d9-7599-446e-8c96-2043fbd46a02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:58.052816 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-0370e9d9-7599-446e-8c96-2043fbd46a02 None None] MFA Rules not processed for user `3e526053edbf4daca2f79303999377b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:58.054026 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0370e9d9-7599-446e-8c96-2043fbd46a02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:58.146219 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 961/1918] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:58 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:46:58.151202 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0778bd12-8c6a-4642-9a9b-7bc7c0f5edb4 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:58.227043 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0778bd12-8c6a-4642-9a9b-7bc7c0f5edb4 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:58.233239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f737038e-623c-46aa-a333-792f965ff9a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:58.233983 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f737038e-623c-46aa-a333-792f965ff9a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:58.234531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f737038e-623c-46aa-a333-792f965ff9a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:58.234928 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f737038e-623c-46aa-a333-792f965ff9a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:58.235689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f737038e-623c-46aa-a333-792f965ff9a8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:58.237836 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f737038e-623c-46aa-a333-792f965ff9a8 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:58.347553 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0370e9d9-7599-446e-8c96-2043fbd46a02 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:58.349274 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0370e9d9-7599-446e-8c96-2043fbd46a02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:58.376581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 958/1919] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:46:58 2022] POST /identity/v3/projects => generated 381 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:58.380289 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0370e9d9-7599-446e-8c96-2043fbd46a02 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:58.384209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f737038e-623c-46aa-a333-792f965ff9a8 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:58.474980 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f737038e-623c-46aa-a333-792f965ff9a8 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:58.484362 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9398db-9837-4e4f-be97-0dfba74f49ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:58.485102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9398db-9837-4e4f-be97-0dfba74f49ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:58.485395 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9398db-9837-4e4f-be97-0dfba74f49ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:58.486904 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9398db-9837-4e4f-be97-0dfba74f49ea admin admin] PATH_INFO: `/v3/users/d13271da32404e2d8d706e2e56bdb054` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:58.489014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9398db-9837-4e4f-be97-0dfba74f49ea admin admin] RBAC: Authorizing `identity:delete_user(user_id=d13271da32404e2d8d706e2e56bdb054)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:58.491572 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1791b6f1-c3d4-415f-aa97-9148ce8ba35e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:58.492422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1791b6f1-c3d4-415f-aa97-9148ce8ba35e admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:58.492531 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9398db-9837-4e4f-be97-0dfba74f49ea admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:58.492706 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1791b6f1-c3d4-415f-aa97-9148ce8ba35e admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:58.492988 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1791b6f1-c3d4-415f-aa97-9148ce8ba35e admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:58.493635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1791b6f1-c3d4-415f-aa97-9148ce8ba35e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:58.496966 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1791b6f1-c3d4-415f-aa97-9148ce8ba35e admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:58.693487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 959/1920] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:46:58 2022] POST /identity/v3/users => generated 297 bytes in 310 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:46:58.698368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1791b6f1-c3d4-415f-aa97-9148ce8ba35e admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:58.741935 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1791b6f1-c3d4-415f-aa97-9148ce8ba35e admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:58.749377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70575ab9-1b03-4f0b-ad98-86f599c082ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:58.753193 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70575ab9-1b03-4f0b-ad98-86f599c082ca admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:58.753193 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70575ab9-1b03-4f0b-ad98-86f599c082ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:58.753193 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70575ab9-1b03-4f0b-ad98-86f599c082ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:58.753193 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70575ab9-1b03-4f0b-ad98-86f599c082ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:58.753655 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70575ab9-1b03-4f0b-ad98-86f599c082ca admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:58.765864 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 960/1921] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:58 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:58.769569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70575ab9-1b03-4f0b-ad98-86f599c082ca admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:58.787887 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70575ab9-1b03-4f0b-ad98-86f599c082ca admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:58.799803 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9559aab-3d10-46a7-9021-c6b27cbe644b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:58.800541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9559aab-3d10-46a7-9021-c6b27cbe644b admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:58.800972 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9559aab-3d10-46a7-9021-c6b27cbe644b admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:58.801301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9559aab-3d10-46a7-9021-c6b27cbe644b admin admin] PATH_INFO: `/v3/projects/fc07a91eb5904acaa15d88b72883af30/users/af8f93807b3f4e85b2fda713063f9454/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:58.824258 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9559aab-3d10-46a7-9021-c6b27cbe644b admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc07a91eb5904acaa15d88b72883af30, user_id=af8f93807b3f4e85b2fda713063f9454, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:58.827545 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9559aab-3d10-46a7-9021-c6b27cbe644b admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:58.842767 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1e9398db-9837-4e4f-be97-0dfba74f49ea admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd13271da32404e2d8d706e2e56bdb054'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:58.843913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1e9398db-9837-4e4f-be97-0dfba74f49ea admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd13271da32404e2d8d706e2e56bdb054'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:58.844184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1e9398db-9837-4e4f-be97-0dfba74f49ea admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd13271da32404e2d8d706e2e56bdb054'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:58.852110 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1e9398db-9837-4e4f-be97-0dfba74f49ea admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd13271da32404e2d8d706e2e56bdb054'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:58.944997 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1e9398db-9837-4e4f-be97-0dfba74f49ea admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd13271da32404e2d8d706e2e56bdb054'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:46:58.954808 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 961/1922] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:58 2022] PUT /identity/v3/projects/fc07a91eb5904acaa15d88b72883af30/users/af8f93807b3f4e85b2fda713063f9454/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:46:58.959107 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9559aab-3d10-46a7-9021-c6b27cbe644b admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:58.961040 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-f9559aab-3d10-46a7-9021-c6b27cbe644b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:58.974640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 962/1923] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:46:58 2022] DELETE /identity/v3/users/d13271da32404e2d8d706e2e56bdb054 => generated 0 bytes in 627 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:58.979090 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d012ba3d-a5ea-454c-8194-e6e2095dbc2c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:58.980394 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d012ba3d-a5ea-454c-8194-e6e2095dbc2c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:58.981236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d012ba3d-a5ea-454c-8194-e6e2095dbc2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:59.017681 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d012ba3d-a5ea-454c-8194-e6e2095dbc2c None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:59.018937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d012ba3d-a5ea-454c-8194-e6e2095dbc2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:59.053760 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9559aab-3d10-46a7-9021-c6b27cbe644b admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:59.147053 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2394aeb-0c8c-4fd2-b44d-b6df20a18dd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:59.147692 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2394aeb-0c8c-4fd2-b44d-b6df20a18dd5 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:59.148141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2394aeb-0c8c-4fd2-b44d-b6df20a18dd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:59.148421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2394aeb-0c8c-4fd2-b44d-b6df20a18dd5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:59.149383 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2394aeb-0c8c-4fd2-b44d-b6df20a18dd5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:59.151088 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 963/1924] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:59.152266 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2394aeb-0c8c-4fd2-b44d-b6df20a18dd5 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:59.164840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 962/1925] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:46:58 2022] GET /identity/v3/roles => generated 1420 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:46:59.169214 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d012ba3d-a5ea-454c-8194-e6e2095dbc2c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:59.181832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d012ba3d-a5ea-454c-8194-e6e2095dbc2c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:59.187763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2394aeb-0c8c-4fd2-b44d-b6df20a18dd5 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:59.187990 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d83d6e-8349-45f6-aacd-8f89d020e64b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:59.188798 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d83d6e-8349-45f6-aacd-8f89d020e64b admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:59.189246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d83d6e-8349-45f6-aacd-8f89d020e64b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:59.189642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d83d6e-8349-45f6-aacd-8f89d020e64b admin admin] PATH_INFO: `/v3/projects/fc07a91eb5904acaa15d88b72883af30/users/af8f93807b3f4e85b2fda713063f9454/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:59.190707 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b2394aeb-0c8c-4fd2-b44d-b6df20a18dd5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:59.192006 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d83d6e-8349-45f6-aacd-8f89d020e64b admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc07a91eb5904acaa15d88b72883af30, user_id=af8f93807b3f4e85b2fda713063f9454, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:59.195637 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d83d6e-8349-45f6-aacd-8f89d020e64b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:59.221819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 964/1926] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:46:59 2022] PUT /identity/v3/projects/fc07a91eb5904acaa15d88b72883af30/users/af8f93807b3f4e85b2fda713063f9454/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:46:59.225279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1e400a-f368-4c45-bba0-1f3cd5df7db1 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:59.225633 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1e400a-f368-4c45-bba0-1f3cd5df7db1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:59.225974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1e400a-f368-4c45-bba0-1f3cd5df7db1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:59.259993 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9c1e400a-f368-4c45-bba0-1f3cd5df7db1 None None] MFA Rules not processed for user `af8f93807b3f4e85b2fda713063f9454`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:46:59.260895 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9c1e400a-f368-4c45-bba0-1f3cd5df7db1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:59.287782 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2394aeb-0c8c-4fd2-b44d-b6df20a18dd5 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:59.350163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd6893d-0178-4681-8d4d-d2522661a7c5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:59.351097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd6893d-0178-4681-8d4d-d2522661a7c5 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:59.351681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd6893d-0178-4681-8d4d-d2522661a7c5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:59.352229 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd6893d-0178-4681-8d4d-d2522661a7c5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:59.397159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:59.412470 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 965/1927] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:46:59 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:46:59.455187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd6893d-0178-4681-8d4d-d2522661a7c5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:59.456694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd6893d-0178-4681-8d4d-d2522661a7c5 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:59.460762 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1e400a-f368-4c45-bba0-1f3cd5df7db1 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:59.463534 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9c1e400a-f368-4c45-bba0-1f3cd5df7db1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:59.469048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 963/1928] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 282 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:46:59.530533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:59.535822 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1e400a-f368-4c45-bba0-1f3cd5df7db1 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:59.540118 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c86057b-2fa0-40c3-93f5-71bae9e97fd4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:59.540772 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c86057b-2fa0-40c3-93f5-71bae9e97fd4 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:59.541053 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c86057b-2fa0-40c3-93f5-71bae9e97fd4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:59.541327 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c86057b-2fa0-40c3-93f5-71bae9e97fd4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:59.551545 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c86057b-2fa0-40c3-93f5-71bae9e97fd4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:59.552619 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c86057b-2fa0-40c3-93f5-71bae9e97fd4 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:46:59.563983 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 966/1929] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:46:59 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:46:59.727455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:46:59.939484 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd6893d-0178-4681-8d4d-d2522661a7c5 service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:46:59.941182 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-9cd6893d-0178-4681-8d4d-d2522661a7c5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:46:59.962328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd6893d-0178-4681-8d4d-d2522661a7c5 service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:46:59.980987 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b732a3-8dea-4218-bad0-7b405c9385ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:46:59.982910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b732a3-8dea-4218-bad0-7b405c9385ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:46:59.983507 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b732a3-8dea-4218-bad0-7b405c9385ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:46:59.983995 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b732a3-8dea-4218-bad0-7b405c9385ab admin admin] PATH_INFO: `/v3/users/353e508f25e14434954de3ee4670f135` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:46:59.994090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b732a3-8dea-4218-bad0-7b405c9385ab admin admin] RBAC: Authorizing `identity:delete_user(user_id=353e508f25e14434954de3ee4670f135)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:46:59.999903 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b732a3-8dea-4218-bad0-7b405c9385ab admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:00.206635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:00.209509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-c6b732a3-8dea-4218-bad0-7b405c9385ab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '353e508f25e14434954de3ee4670f135'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:00.228587 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-c6b732a3-8dea-4218-bad0-7b405c9385ab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '353e508f25e14434954de3ee4670f135'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:00.229421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-c6b732a3-8dea-4218-bad0-7b405c9385ab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '353e508f25e14434954de3ee4670f135'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:00.255352 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c86057b-2fa0-40c3-93f5-71bae9e97fd4 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:00.256886 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3c86057b-2fa0-40c3-93f5-71bae9e97fd4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:00.280227 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c86057b-2fa0-40c3-93f5-71bae9e97fd4 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:00.287665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc20ffbf-aa8f-4ab0-9c5d-c6f89e3ccc4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:00.288840 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc20ffbf-aa8f-4ab0-9c5d-c6f89e3ccc4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:00.289323 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc20ffbf-aa8f-4ab0-9c5d-c6f89e3ccc4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:00.289855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc20ffbf-aa8f-4ab0-9c5d-c6f89e3ccc4d admin admin] PATH_INFO: `/v3/projects/02501f755ba54c339a3b8808b8f0820e` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:00.292433 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc20ffbf-aa8f-4ab0-9c5d-c6f89e3ccc4d admin admin] RBAC: Authorizing `identity:delete_project(project_id=02501f755ba54c339a3b8808b8f0820e)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:00.295569 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc20ffbf-aa8f-4ab0-9c5d-c6f89e3ccc4d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:00.299270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-c6b732a3-8dea-4218-bad0-7b405c9385ab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '353e508f25e14434954de3ee4670f135'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:00.315218 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-c6b732a3-8dea-4218-bad0-7b405c9385ab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '353e508f25e14434954de3ee4670f135'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:00.328889 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:00.331161 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 964/1930] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:46:59 2022] DELETE /identity/v3/users/353e508f25e14434954de3ee4670f135 => generated 0 bytes in 394 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:00.335908 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e507f3-b5a8-4fb2-a259-cf23c9ff9ae4 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:00.337580 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e507f3-b5a8-4fb2-a259-cf23c9ff9ae4 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:00.338713 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e507f3-b5a8-4fb2-a259-cf23c9ff9ae4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:00.372117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-f3e507f3-b5a8-4fb2-a259-cf23c9ff9ae4 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:00.374560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-f3e507f3-b5a8-4fb2-a259-cf23c9ff9ae4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:00.441582 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 965/1931] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:47:00.456620 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e507f3-b5a8-4fb2-a259-cf23c9ff9ae4 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:00.460127 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-f3e507f3-b5a8-4fb2-a259-cf23c9ff9ae4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:00.515106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cc20ffbf-aa8f-4ab0-9c5d-c6f89e3ccc4d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '02501f755ba54c339a3b8808b8f0820e', 'request_id': 'req-cc20ffbf-aa8f-4ab0-9c5d-c6f89e3ccc4d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:00.537469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cc20ffbf-aa8f-4ab0-9c5d-c6f89e3ccc4d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '02501f755ba54c339a3b8808b8f0820e', 'request_id': 'req-cc20ffbf-aa8f-4ab0-9c5d-c6f89e3ccc4d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:00.550424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e507f3-b5a8-4fb2-a259-cf23c9ff9ae4 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:00.553794 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cc20ffbf-aa8f-4ab0-9c5d-c6f89e3ccc4d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '02501f755ba54c339a3b8808b8f0820e', 'request_id': 'req-cc20ffbf-aa8f-4ab0-9c5d-c6f89e3ccc4d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:00.611064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19799ba-5dab-4fd5-8b57-c91d42ee1b2a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:00.611736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19799ba-5dab-4fd5-8b57-c91d42ee1b2a service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:00.612015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19799ba-5dab-4fd5-8b57-c91d42ee1b2a service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:00.612284 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19799ba-5dab-4fd5-8b57-c91d42ee1b2a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:00.677980 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19799ba-5dab-4fd5-8b57-c91d42ee1b2a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:00.679506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19799ba-5dab-4fd5-8b57-c91d42ee1b2a service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:00.689316 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cc20ffbf-aa8f-4ab0-9c5d-c6f89e3ccc4d admin admin] The token cache is being invalidate because project 02501f755ba54c339a3b8808b8f0820e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:00.689839 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cc20ffbf-aa8f-4ab0-9c5d-c6f89e3ccc4d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:00.690706 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cc20ffbf-aa8f-4ab0-9c5d-c6f89e3ccc4d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:00.691204 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 966/1932] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 236 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:00.692058 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19799ba-5dab-4fd5-8b57-c91d42ee1b2a service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:00.692407 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 967/1933] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:00 2022] DELETE /identity/v3/projects/02501f755ba54c339a3b8808b8f0820e => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:47:00.696494 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c19799ba-5dab-4fd5-8b57-c91d42ee1b2a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:00.697726 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc20ffbf-aa8f-4ab0-9c5d-c6f89e3ccc4d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:00.700000 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-cc20ffbf-aa8f-4ab0-9c5d-c6f89e3ccc4d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:00.718641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19799ba-5dab-4fd5-8b57-c91d42ee1b2a service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:00.728138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc20ffbf-aa8f-4ab0-9c5d-c6f89e3ccc4d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:00.731203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1b336f-40b3-459e-bd2d-5da594f3c00c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:00.731986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1b336f-40b3-459e-bd2d-5da594f3c00c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:00.732359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1b336f-40b3-459e-bd2d-5da594f3c00c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:00.732697 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1b336f-40b3-459e-bd2d-5da594f3c00c admin admin] PATH_INFO: `/v3/users/3e526053edbf4daca2f79303999377b1` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:00.734819 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1b336f-40b3-459e-bd2d-5da594f3c00c admin admin] RBAC: Authorizing `identity:delete_user(user_id=3e526053edbf4daca2f79303999377b1)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:00.749912 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1b336f-40b3-459e-bd2d-5da594f3c00c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:00.750058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5a94a4-1d04-4b66-bb8e-6b7ce7e3b925 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:00.750058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5a94a4-1d04-4b66-bb8e-6b7ce7e3b925 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:00.750058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5a94a4-1d04-4b66-bb8e-6b7ce7e3b925 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:00.750058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5a94a4-1d04-4b66-bb8e-6b7ce7e3b925 admin admin] PATH_INFO: `/v3/users/46b1d4c8ad2e41d194789555f09727a5` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:00.753818 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5a94a4-1d04-4b66-bb8e-6b7ce7e3b925 admin admin] RBAC: Authorizing `identity:delete_user(user_id=46b1d4c8ad2e41d194789555f09727a5)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:00.757608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5a94a4-1d04-4b66-bb8e-6b7ce7e3b925 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:00.978772 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8e5a94a4-1d04-4b66-bb8e-6b7ce7e3b925 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '46b1d4c8ad2e41d194789555f09727a5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:00.979682 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8e5a94a4-1d04-4b66-bb8e-6b7ce7e3b925 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '46b1d4c8ad2e41d194789555f09727a5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:00.980493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8e5a94a4-1d04-4b66-bb8e-6b7ce7e3b925 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '46b1d4c8ad2e41d194789555f09727a5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:01.002484 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8e5a94a4-1d04-4b66-bb8e-6b7ce7e3b925 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '46b1d4c8ad2e41d194789555f09727a5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:01.019669 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-ae1b336f-40b3-459e-bd2d-5da594f3c00c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3e526053edbf4daca2f79303999377b1'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:01.038374 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-ae1b336f-40b3-459e-bd2d-5da594f3c00c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3e526053edbf4daca2f79303999377b1'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:01.039963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-ae1b336f-40b3-459e-bd2d-5da594f3c00c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3e526053edbf4daca2f79303999377b1'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:01.094473 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8e5a94a4-1d04-4b66-bb8e-6b7ce7e3b925 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '46b1d4c8ad2e41d194789555f09727a5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:01.124941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:01.130753 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 968/1934] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:00 2022] DELETE /identity/v3/users/46b1d4c8ad2e41d194789555f09727a5 => generated 0 bytes in 434 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:01.212730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-ae1b336f-40b3-459e-bd2d-5da594f3c00c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3e526053edbf4daca2f79303999377b1'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:01.224457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-ae1b336f-40b3-459e-bd2d-5da594f3c00c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3e526053edbf4daca2f79303999377b1'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:01.228424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5a94a4-1d04-4b66-bb8e-6b7ce7e3b925 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:01.230035 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8e5a94a4-1d04-4b66-bb8e-6b7ce7e3b925 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:01.254102 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 967/1935] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:00 2022] DELETE /identity/v3/users/3e526053edbf4daca2f79303999377b1 => generated 0 bytes in 562 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:01.255862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1b336f-40b3-459e-bd2d-5da594f3c00c admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:01.264245 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ae1b336f-40b3-459e-bd2d-5da594f3c00c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:01.356511 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5a94a4-1d04-4b66-bb8e-6b7ce7e3b925 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:01.357976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1b336f-40b3-459e-bd2d-5da594f3c00c admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:01.362641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99de06e-6109-4d5d-8c38-adb0426528ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:01.363341 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99de06e-6109-4d5d-8c38-adb0426528ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:01.363628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99de06e-6109-4d5d-8c38-adb0426528ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:01.366721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99de06e-6109-4d5d-8c38-adb0426528ec admin admin] PATH_INFO: `/v3/users/851aecca45a64b058441375f78c4e2b1` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:01.366920 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9935ed37-7049-44a0-b5d0-be1f7cc004db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:01.367258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9935ed37-7049-44a0-b5d0-be1f7cc004db admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:01.367555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9935ed37-7049-44a0-b5d0-be1f7cc004db admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:01.367833 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9935ed37-7049-44a0-b5d0-be1f7cc004db admin admin] PATH_INFO: `/v3/projects/be14f805af7a4180aea48756132410a5` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:01.368962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99de06e-6109-4d5d-8c38-adb0426528ec admin admin] RBAC: Authorizing `identity:delete_user(user_id=851aecca45a64b058441375f78c4e2b1)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:01.371298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99de06e-6109-4d5d-8c38-adb0426528ec admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:01.371428 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9935ed37-7049-44a0-b5d0-be1f7cc004db admin admin] RBAC: Authorizing `identity:delete_project(project_id=be14f805af7a4180aea48756132410a5)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:01.374756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9935ed37-7049-44a0-b5d0-be1f7cc004db admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:01.385493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=5,keystone:SELECT=93,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:01.585721 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9935ed37-7049-44a0-b5d0-be1f7cc004db admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'be14f805af7a4180aea48756132410a5', 'request_id': 'req-9935ed37-7049-44a0-b5d0-be1f7cc004db'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:01.594390 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9935ed37-7049-44a0-b5d0-be1f7cc004db admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'be14f805af7a4180aea48756132410a5', 'request_id': 'req-9935ed37-7049-44a0-b5d0-be1f7cc004db'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:01.602654 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9935ed37-7049-44a0-b5d0-be1f7cc004db admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'be14f805af7a4180aea48756132410a5', 'request_id': 'req-9935ed37-7049-44a0-b5d0-be1f7cc004db'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:01.606641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a99de06e-6109-4d5d-8c38-adb0426528ec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '851aecca45a64b058441375f78c4e2b1'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:01.645473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a99de06e-6109-4d5d-8c38-adb0426528ec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '851aecca45a64b058441375f78c4e2b1'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:01.647247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a99de06e-6109-4d5d-8c38-adb0426528ec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '851aecca45a64b058441375f78c4e2b1'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:01.710906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9935ed37-7049-44a0-b5d0-be1f7cc004db admin admin] The token cache is being invalidate because project be14f805af7a4180aea48756132410a5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:01.711204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9935ed37-7049-44a0-b5d0-be1f7cc004db admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:01.713741 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9935ed37-7049-44a0-b5d0-be1f7cc004db admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:01.713741 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 969/1936] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:01 2022] DELETE /identity/v3/projects/be14f805af7a4180aea48756132410a5 => generated 0 bytes in 486 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:01.714721 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fce3369-fc69-497b-93f2-b15c03cce19c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:01.715005 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fce3369-fc69-497b-93f2-b15c03cce19c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:01.715278 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fce3369-fc69-497b-93f2-b15c03cce19c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:01.735276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-8fce3369-fc69-497b-93f2-b15c03cce19c None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:01.860913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8fce3369-fc69-497b-93f2-b15c03cce19c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:01.860913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 970/1937] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:01.860913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fce3369-fc69-497b-93f2-b15c03cce19c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:01.860913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8fce3369-fc69-497b-93f2-b15c03cce19c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:01.860913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fce3369-fc69-497b-93f2-b15c03cce19c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:01.862182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2286c5ec-a1ae-4c08-9746-42b01fd5f33b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:01.862182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2286c5ec-a1ae-4c08-9746-42b01fd5f33b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:01.862182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2286c5ec-a1ae-4c08-9746-42b01fd5f33b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:01.862182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2286c5ec-a1ae-4c08-9746-42b01fd5f33b admin admin] PATH_INFO: `/v3/projects/f3ce98276abc499fbd2a329e523fa124` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:01.863342 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2286c5ec-a1ae-4c08-9746-42b01fd5f33b admin admin] RBAC: Authorizing `identity:delete_project(project_id=f3ce98276abc499fbd2a329e523fa124)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:01.863342 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2286c5ec-a1ae-4c08-9746-42b01fd5f33b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:01.864232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a99de06e-6109-4d5d-8c38-adb0426528ec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '851aecca45a64b058441375f78c4e2b1'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:01.864232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a99de06e-6109-4d5d-8c38-adb0426528ec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '851aecca45a64b058441375f78c4e2b1'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:01.868296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 968/1938] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:01 2022] DELETE /identity/v3/users/851aecca45a64b058441375f78c4e2b1 => generated 0 bytes in 610 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:01.869254 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99de06e-6109-4d5d-8c38-adb0426528ec admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:01.871068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a99de06e-6109-4d5d-8c38-adb0426528ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:01.938147 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99de06e-6109-4d5d-8c38-adb0426528ec admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:01.942790 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02217a19-5442-4e62-9f45-37d7f3f5592a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:01.943450 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02217a19-5442-4e62-9f45-37d7f3f5592a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:01.943729 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02217a19-5442-4e62-9f45-37d7f3f5592a admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:01.944001 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02217a19-5442-4e62-9f45-37d7f3f5592a admin admin] PATH_INFO: `/v3/users/ec2bd6ae792143c094f7657c6d6ec745` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:01.945107 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02217a19-5442-4e62-9f45-37d7f3f5592a admin admin] RBAC: Authorizing `identity:delete_user(user_id=ec2bd6ae792143c094f7657c6d6ec745)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:01.947093 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02217a19-5442-4e62-9f45-37d7f3f5592a admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:01.975754 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2286c5ec-a1ae-4c08-9746-42b01fd5f33b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f3ce98276abc499fbd2a329e523fa124', 'request_id': 'req-2286c5ec-a1ae-4c08-9746-42b01fd5f33b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:01.981178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2286c5ec-a1ae-4c08-9746-42b01fd5f33b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f3ce98276abc499fbd2a329e523fa124', 'request_id': 'req-2286c5ec-a1ae-4c08-9746-42b01fd5f33b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:01.987900 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2286c5ec-a1ae-4c08-9746-42b01fd5f33b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f3ce98276abc499fbd2a329e523fa124', 'request_id': 'req-2286c5ec-a1ae-4c08-9746-42b01fd5f33b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:02.069162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2286c5ec-a1ae-4c08-9746-42b01fd5f33b admin admin] The token cache is being invalidate because project f3ce98276abc499fbd2a329e523fa124 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:02.070456 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2286c5ec-a1ae-4c08-9746-42b01fd5f33b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:02.070456 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2286c5ec-a1ae-4c08-9746-42b01fd5f33b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:02.071725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 971/1939] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:01 2022] DELETE /identity/v3/projects/f3ce98276abc499fbd2a329e523fa124 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:02.072538 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2286c5ec-a1ae-4c08-9746-42b01fd5f33b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:02.073894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2286c5ec-a1ae-4c08-9746-42b01fd5f33b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:02.124253 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-02217a19-5442-4e62-9f45-37d7f3f5592a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ec2bd6ae792143c094f7657c6d6ec745'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:02.130864 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2286c5ec-a1ae-4c08-9746-42b01fd5f33b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:02.134393 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aec288a-e821-4b88-b70a-53946469e971 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:02.135043 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aec288a-e821-4b88-b70a-53946469e971 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:02.135306 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aec288a-e821-4b88-b70a-53946469e971 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:02.135584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aec288a-e821-4b88-b70a-53946469e971 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:02.136720 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-02217a19-5442-4e62-9f45-37d7f3f5592a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ec2bd6ae792143c094f7657c6d6ec745'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:02.137256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3aec288a-e821-4b88-b70a-53946469e971 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:02.137514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-02217a19-5442-4e62-9f45-37d7f3f5592a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ec2bd6ae792143c094f7657c6d6ec745'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:02.159069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:02.198616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aec288a-e821-4b88-b70a-53946469e971 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:02.200153 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aec288a-e821-4b88-b70a-53946469e971 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:02.209618 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 972/1940] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:02.210949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aff4640-33fb-4e8a-a6ab-2c41a5423ff5 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:02.211238 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aff4640-33fb-4e8a-a6ab-2c41a5423ff5 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:02.211513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aff4640-33fb-4e8a-a6ab-2c41a5423ff5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:02.227979 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-02217a19-5442-4e62-9f45-37d7f3f5592a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ec2bd6ae792143c094f7657c6d6ec745'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:02.232080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-5aff4640-33fb-4e8a-a6ab-2c41a5423ff5 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:02.233262 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5aff4640-33fb-4e8a-a6ab-2c41a5423ff5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:02.238287 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-02217a19-5442-4e62-9f45-37d7f3f5592a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ec2bd6ae792143c094f7657c6d6ec745'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:02.241659 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 973/1941] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:02.242470 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aff4640-33fb-4e8a-a6ab-2c41a5423ff5 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:02.243823 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5aff4640-33fb-4e8a-a6ab-2c41a5423ff5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:02.246492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 969/1942] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:01 2022] DELETE /identity/v3/users/ec2bd6ae792143c094f7657c6d6ec745 => generated 0 bytes in 378 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:02.248075 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba35356d-69e1-4413-af42-e0300bc57b0c None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:02.248330 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba35356d-69e1-4413-af42-e0300bc57b0c None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:02.248605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba35356d-69e1-4413-af42-e0300bc57b0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:02.260997 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aff4640-33fb-4e8a-a6ab-2c41a5423ff5 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:02.280052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-ba35356d-69e1-4413-af42-e0300bc57b0c None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:02.282010 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ba35356d-69e1-4413-af42-e0300bc57b0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:02.283020 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:02.336407 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56aaab4a-1f17-4322-8458-1db73ec7d77d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:02.337609 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56aaab4a-1f17-4322-8458-1db73ec7d77d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:02.338102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56aaab4a-1f17-4322-8458-1db73ec7d77d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:02.338542 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56aaab4a-1f17-4322-8458-1db73ec7d77d admin admin] PATH_INFO: `/v3/users/5a978f07591d4b78b248c27546a92b40` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:02.340469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56aaab4a-1f17-4322-8458-1db73ec7d77d admin admin] RBAC: Authorizing `identity:delete_user(user_id=5a978f07591d4b78b248c27546a92b40)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:02.343293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56aaab4a-1f17-4322-8458-1db73ec7d77d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:02.384876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 970/1943] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:02.384876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba35356d-69e1-4413-af42-e0300bc57b0c None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:02.384876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ba35356d-69e1-4413-af42-e0300bc57b0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:02.441487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba35356d-69e1-4413-af42-e0300bc57b0c None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:02.447605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c93dc3a-7e6c-4b5e-90a0-ee71e9ae3956 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:02.448237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c93dc3a-7e6c-4b5e-90a0-ee71e9ae3956 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:02.448516 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c93dc3a-7e6c-4b5e-90a0-ee71e9ae3956 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:02.448787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c93dc3a-7e6c-4b5e-90a0-ee71e9ae3956 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:02.458443 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c93dc3a-7e6c-4b5e-90a0-ee71e9ae3956 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:02.460503 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c93dc3a-7e6c-4b5e-90a0-ee71e9ae3956 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:02.470305 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 971/1944] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:02.471624 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3366d5ed-ac8a-4b28-b648-536748b0905b None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:02.471909 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3366d5ed-ac8a-4b28-b648-536748b0905b None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:02.472179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3366d5ed-ac8a-4b28-b648-536748b0905b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:02.493151 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-3366d5ed-ac8a-4b28-b648-536748b0905b None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:02.495345 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-3366d5ed-ac8a-4b28-b648-536748b0905b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:02.503512 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 972/1945] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:02.507878 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3366d5ed-ac8a-4b28-b648-536748b0905b None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:02.518240 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3366d5ed-ac8a-4b28-b648-536748b0905b None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:02.523315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dc89ed-a419-43ad-a6ec-88f2a346f92a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:02.523992 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dc89ed-a419-43ad-a6ec-88f2a346f92a admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:02.524338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dc89ed-a419-43ad-a6ec-88f2a346f92a admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:02.524689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dc89ed-a419-43ad-a6ec-88f2a346f92a admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:02.525467 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dc89ed-a419-43ad-a6ec-88f2a346f92a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:47:02.525842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dc89ed-a419-43ad-a6ec-88f2a346f92a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:02.527767 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dc89ed-a419-43ad-a6ec-88f2a346f92a admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:02.544314 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 973/1946] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:47:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:47:02.549975 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dc89ed-a419-43ad-a6ec-88f2a346f92a admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:02.563273 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dc89ed-a419-43ad-a6ec-88f2a346f92a admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:02.569732 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb1b0dc-e459-41e9-80f7-1a44042aecbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:02.570417 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb1b0dc-e459-41e9-80f7-1a44042aecbf admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:02.570837 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb1b0dc-e459-41e9-80f7-1a44042aecbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:02.571180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb1b0dc-e459-41e9-80f7-1a44042aecbf admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:02.571966 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb1b0dc-e459-41e9-80f7-1a44042aecbf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:02.573946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb1b0dc-e459-41e9-80f7-1a44042aecbf admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:02.590533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-56aaab4a-1f17-4322-8458-1db73ec7d77d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5a978f07591d4b78b248c27546a92b40'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:02.591314 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-56aaab4a-1f17-4322-8458-1db73ec7d77d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5a978f07591d4b78b248c27546a92b40'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:02.592524 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-56aaab4a-1f17-4322-8458-1db73ec7d77d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5a978f07591d4b78b248c27546a92b40'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:02.600031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-56aaab4a-1f17-4322-8458-1db73ec7d77d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5a978f07591d4b78b248c27546a92b40'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:02.656753 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 974/1947] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:02 2022] POST /identity/v3/projects => generated 401 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:02.660854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb1b0dc-e459-41e9-80f7-1a44042aecbf admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:02.662878 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-5bb1b0dc-e459-41e9-80f7-1a44042aecbf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:02.671167 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-56aaab4a-1f17-4322-8458-1db73ec7d77d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5a978f07591d4b78b248c27546a92b40'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:02.714708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 974/1948] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:02 2022] DELETE /identity/v3/users/5a978f07591d4b78b248c27546a92b40 => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:02.719051 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c7a6a0-55bd-4368-9dc6-e3318baaf9e7 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:02.719666 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c7a6a0-55bd-4368-9dc6-e3318baaf9e7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:02.721353 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c7a6a0-55bd-4368-9dc6-e3318baaf9e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:02.747552 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d3c7a6a0-55bd-4368-9dc6-e3318baaf9e7 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:02.752693 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d3c7a6a0-55bd-4368-9dc6-e3318baaf9e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:02.798747 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb1b0dc-e459-41e9-80f7-1a44042aecbf admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:02.825286 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 975/1949] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:02.845194 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c7a6a0-55bd-4368-9dc6-e3318baaf9e7 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:02.847852 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d3c7a6a0-55bd-4368-9dc6-e3318baaf9e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:02.869287 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a10661d-250f-4b1b-9410-c9ee204b43a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:02.870809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a10661d-250f-4b1b-9410-c9ee204b43a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:02.871099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a10661d-250f-4b1b-9410-c9ee204b43a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:02.871498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a10661d-250f-4b1b-9410-c9ee204b43a1 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:02.872148 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a10661d-250f-4b1b-9410-c9ee204b43a1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:02.874794 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a10661d-250f-4b1b-9410-c9ee204b43a1 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:02.898756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:02.927547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c7a6a0-55bd-4368-9dc6-e3318baaf9e7 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:02.931169 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759cff40-96e4-472d-8356-3eaade3c762c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:02.931775 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759cff40-96e4-472d-8356-3eaade3c762c service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:02.932052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759cff40-96e4-472d-8356-3eaade3c762c service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:02.932327 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759cff40-96e4-472d-8356-3eaade3c762c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:02.941749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759cff40-96e4-472d-8356-3eaade3c762c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:02.943253 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759cff40-96e4-472d-8356-3eaade3c762c service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:02.950333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 975/1950] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:02 2022] POST /identity/v3/users => generated 301 bytes in 290 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:02.953018 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a10661d-250f-4b1b-9410-c9ee204b43a1 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:02.953568 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 976/1951] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:02.954389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759cff40-96e4-472d-8356-3eaade3c762c service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:02.955763 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-759cff40-96e4-472d-8356-3eaade3c762c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:02.956328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-8a10661d-250f-4b1b-9410-c9ee204b43a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:02.968702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759cff40-96e4-472d-8356-3eaade3c762c service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:02.969401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a10661d-250f-4b1b-9410-c9ee204b43a1 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:02.973536 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d57b15-6e82-4a15-9a2e-be8625c59c2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:02.974169 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d57b15-6e82-4a15-9a2e-be8625c59c2e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:02.974444 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d57b15-6e82-4a15-9a2e-be8625c59c2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:02.974750 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d57b15-6e82-4a15-9a2e-be8625c59c2e admin admin] PATH_INFO: `/v3/projects/a0b1b0614863438c99c08917ec713768` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:02.975253 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5076d4a9-da8a-4590-902b-c84cf2f3c83d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:02.975951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5076d4a9-da8a-4590-902b-c84cf2f3c83d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:02.976075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d57b15-6e82-4a15-9a2e-be8625c59c2e admin admin] RBAC: Authorizing `identity:delete_project(project_id=a0b1b0614863438c99c08917ec713768)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:02.976370 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5076d4a9-da8a-4590-902b-c84cf2f3c83d admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:02.976709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5076d4a9-da8a-4590-902b-c84cf2f3c83d admin admin] PATH_INFO: `/v3/projects/3ab149fcc9c3411f8b3e08ebe2907109` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:02.977928 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5076d4a9-da8a-4590-902b-c84cf2f3c83d admin admin] RBAC: Authorizing `identity:delete_project(project_id=3ab149fcc9c3411f8b3e08ebe2907109)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:02.978036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d57b15-6e82-4a15-9a2e-be8625c59c2e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:02.980340 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5076d4a9-da8a-4590-902b-c84cf2f3c83d admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:03.007741 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:03.127969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5076d4a9-da8a-4590-902b-c84cf2f3c83d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3ab149fcc9c3411f8b3e08ebe2907109', 'request_id': 'req-5076d4a9-da8a-4590-902b-c84cf2f3c83d'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.134811 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-62d57b15-6e82-4a15-9a2e-be8625c59c2e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a0b1b0614863438c99c08917ec713768', 'request_id': 'req-62d57b15-6e82-4a15-9a2e-be8625c59c2e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.136274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5076d4a9-da8a-4590-902b-c84cf2f3c83d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3ab149fcc9c3411f8b3e08ebe2907109', 'request_id': 'req-5076d4a9-da8a-4590-902b-c84cf2f3c83d'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.142205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-62d57b15-6e82-4a15-9a2e-be8625c59c2e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a0b1b0614863438c99c08917ec713768', 'request_id': 'req-62d57b15-6e82-4a15-9a2e-be8625c59c2e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.143975 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5076d4a9-da8a-4590-902b-c84cf2f3c83d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3ab149fcc9c3411f8b3e08ebe2907109', 'request_id': 'req-5076d4a9-da8a-4590-902b-c84cf2f3c83d'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.149407 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-62d57b15-6e82-4a15-9a2e-be8625c59c2e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a0b1b0614863438c99c08917ec713768', 'request_id': 'req-62d57b15-6e82-4a15-9a2e-be8625c59c2e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.236179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5076d4a9-da8a-4590-902b-c84cf2f3c83d admin admin] The token cache is being invalidate because project 3ab149fcc9c3411f8b3e08ebe2907109 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:03.239339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5076d4a9-da8a-4590-902b-c84cf2f3c83d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.239339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5076d4a9-da8a-4590-902b-c84cf2f3c83d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.241704 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 976/1952] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:02 2022] DELETE /identity/v3/projects/3ab149fcc9c3411f8b3e08ebe2907109 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:03.242510 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5076d4a9-da8a-4590-902b-c84cf2f3c83d admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:03.243884 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-5076d4a9-da8a-4590-902b-c84cf2f3c83d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:03.290269 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-62d57b15-6e82-4a15-9a2e-be8625c59c2e admin admin] The token cache is being invalidate because project a0b1b0614863438c99c08917ec713768 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:03.290857 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-62d57b15-6e82-4a15-9a2e-be8625c59c2e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.291360 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-62d57b15-6e82-4a15-9a2e-be8625c59c2e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.292886 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 977/1953] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:02 2022] DELETE /identity/v3/projects/a0b1b0614863438c99c08917ec713768 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:03.293717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d57b15-6e82-4a15-9a2e-be8625c59c2e admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:03.295622 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-62d57b15-6e82-4a15-9a2e-be8625c59c2e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:03.308072 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5076d4a9-da8a-4590-902b-c84cf2f3c83d admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:03.358343 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d57b15-6e82-4a15-9a2e-be8625c59c2e admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:03.365365 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60dd26f0-d19b-408c-94b2-406a90bcc158 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:03.366219 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dd26f0-d19b-408c-94b2-406a90bcc158 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:03.366507 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dd26f0-d19b-408c-94b2-406a90bcc158 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:03.366789 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dd26f0-d19b-408c-94b2-406a90bcc158 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:03.367428 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60dd26f0-d19b-408c-94b2-406a90bcc158 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:03.368295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a895464-9146-403c-b1c3-041e5647bc5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:03.368948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a895464-9146-403c-b1c3-041e5647bc5d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:03.369068 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60dd26f0-d19b-408c-94b2-406a90bcc158 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:03.369446 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a895464-9146-403c-b1c3-041e5647bc5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:03.369626 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a895464-9146-403c-b1c3-041e5647bc5d admin admin] PATH_INFO: `/v3/projects/e642241b30124f0cbf0fe8d9d67d8f47` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:03.371166 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a895464-9146-403c-b1c3-041e5647bc5d admin admin] RBAC: Authorizing `identity:delete_project(project_id=e642241b30124f0cbf0fe8d9d67d8f47)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:03.373641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a895464-9146-403c-b1c3-041e5647bc5d admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:03.380986 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 978/1954] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:03 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:03.382132 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60dd26f0-d19b-408c-94b2-406a90bcc158 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:03.385293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-60dd26f0-d19b-408c-94b2-406a90bcc158 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:03.400051 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60dd26f0-d19b-408c-94b2-406a90bcc158 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:03.404898 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a164c7f-c5b2-404f-a934-a350b22c0a78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:03.405531 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a164c7f-c5b2-404f-a934-a350b22c0a78 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:03.405814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a164c7f-c5b2-404f-a934-a350b22c0a78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:03.408963 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a164c7f-c5b2-404f-a934-a350b22c0a78 admin admin] PATH_INFO: `/v3/users/af8f93807b3f4e85b2fda713063f9454` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:03.408963 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a164c7f-c5b2-404f-a934-a350b22c0a78 admin admin] RBAC: Authorizing `identity:delete_user(user_id=af8f93807b3f4e85b2fda713063f9454)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:03.409270 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a164c7f-c5b2-404f-a934-a350b22c0a78 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:03.493768 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7a895464-9146-403c-b1c3-041e5647bc5d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e642241b30124f0cbf0fe8d9d67d8f47', 'request_id': 'req-7a895464-9146-403c-b1c3-041e5647bc5d'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.501066 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7a895464-9146-403c-b1c3-041e5647bc5d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e642241b30124f0cbf0fe8d9d67d8f47', 'request_id': 'req-7a895464-9146-403c-b1c3-041e5647bc5d'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.506838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7a895464-9146-403c-b1c3-041e5647bc5d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e642241b30124f0cbf0fe8d9d67d8f47', 'request_id': 'req-7a895464-9146-403c-b1c3-041e5647bc5d'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.553747 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:03.588693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7a895464-9146-403c-b1c3-041e5647bc5d admin admin] The token cache is being invalidate because project e642241b30124f0cbf0fe8d9d67d8f47 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:03.588991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7a895464-9146-403c-b1c3-041e5647bc5d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.589654 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7a895464-9146-403c-b1c3-041e5647bc5d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.594538 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 977/1955] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:03 2022] DELETE /identity/v3/projects/e642241b30124f0cbf0fe8d9d67d8f47 => generated 0 bytes in 349 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:03.594538 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a23a13-531b-432d-b303-5c10b25aa583 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:03.594538 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a23a13-531b-432d-b303-5c10b25aa583 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:03.594538 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a23a13-531b-432d-b303-5c10b25aa583 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:03.601774 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1a164c7f-c5b2-404f-a934-a350b22c0a78 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af8f93807b3f4e85b2fda713063f9454'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.602577 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1a164c7f-c5b2-404f-a934-a350b22c0a78 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af8f93807b3f4e85b2fda713063f9454'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.604100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1a164c7f-c5b2-404f-a934-a350b22c0a78 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af8f93807b3f4e85b2fda713063f9454'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.611954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:03.614193 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1a164c7f-c5b2-404f-a934-a350b22c0a78 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af8f93807b3f4e85b2fda713063f9454'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.615373 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-13a23a13-531b-432d-b303-5c10b25aa583 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:03.616649 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-13a23a13-531b-432d-b303-5c10b25aa583 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:03.677841 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 978/1956] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:03.678693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a23a13-531b-432d-b303-5c10b25aa583 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:03.679941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-13a23a13-531b-432d-b303-5c10b25aa583 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:03.691421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a23a13-531b-432d-b303-5c10b25aa583 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:03.696073 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496cce25-fc0f-4994-952e-a58718b82dc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:03.696677 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496cce25-fc0f-4994-952e-a58718b82dc6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:03.696958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496cce25-fc0f-4994-952e-a58718b82dc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:03.697231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496cce25-fc0f-4994-952e-a58718b82dc6 admin admin] PATH_INFO: `/v3/projects/d38d518c11a34ffc8921d4c241b8ce06` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:03.698324 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496cce25-fc0f-4994-952e-a58718b82dc6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d38d518c11a34ffc8921d4c241b8ce06)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:03.700276 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496cce25-fc0f-4994-952e-a58718b82dc6 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:03.713069 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1a164c7f-c5b2-404f-a934-a350b22c0a78 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af8f93807b3f4e85b2fda713063f9454'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.731858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 979/1957] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:03 2022] DELETE /identity/v3/users/af8f93807b3f4e85b2fda713063f9454 => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:03.735495 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a164c7f-c5b2-404f-a934-a350b22c0a78 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:03.735495 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1a164c7f-c5b2-404f-a934-a350b22c0a78 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:03.800113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a164c7f-c5b2-404f-a934-a350b22c0a78 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:03.805660 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522aee75-7e93-48f0-9864-a5bfc58c5a3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:03.806279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522aee75-7e93-48f0-9864-a5bfc58c5a3a admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:03.806565 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522aee75-7e93-48f0-9864-a5bfc58c5a3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:03.806913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522aee75-7e93-48f0-9864-a5bfc58c5a3a admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:03.807517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522aee75-7e93-48f0-9864-a5bfc58c5a3a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:03.809139 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522aee75-7e93-48f0-9864-a5bfc58c5a3a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:03.820438 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 980/1958] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:03 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:03.821264 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522aee75-7e93-48f0-9864-a5bfc58c5a3a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:03.823125 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-522aee75-7e93-48f0-9864-a5bfc58c5a3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:03.831242 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-496cce25-fc0f-4994-952e-a58718b82dc6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd38d518c11a34ffc8921d4c241b8ce06', 'request_id': 'req-496cce25-fc0f-4994-952e-a58718b82dc6'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.835902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522aee75-7e93-48f0-9864-a5bfc58c5a3a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:03.836887 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-496cce25-fc0f-4994-952e-a58718b82dc6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd38d518c11a34ffc8921d4c241b8ce06', 'request_id': 'req-496cce25-fc0f-4994-952e-a58718b82dc6'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.844151 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-496cce25-fc0f-4994-952e-a58718b82dc6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd38d518c11a34ffc8921d4c241b8ce06', 'request_id': 'req-496cce25-fc0f-4994-952e-a58718b82dc6'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.894732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328a7821-cb19-4cf1-a802-c7854b336d51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:03.895511 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328a7821-cb19-4cf1-a802-c7854b336d51 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:03.895793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328a7821-cb19-4cf1-a802-c7854b336d51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:03.896066 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328a7821-cb19-4cf1-a802-c7854b336d51 admin admin] PATH_INFO: `/v3/users/7eb49e88af3e422581776a81236f31af` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:03.897238 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328a7821-cb19-4cf1-a802-c7854b336d51 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7eb49e88af3e422581776a81236f31af)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:03.899262 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328a7821-cb19-4cf1-a802-c7854b336d51 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:03.925584 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-496cce25-fc0f-4994-952e-a58718b82dc6 admin admin] The token cache is being invalidate because project d38d518c11a34ffc8921d4c241b8ce06 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:03.925955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-496cce25-fc0f-4994-952e-a58718b82dc6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.928911 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-496cce25-fc0f-4994-952e-a58718b82dc6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:03.928911 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 979/1959] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:03 2022] DELETE /identity/v3/projects/d38d518c11a34ffc8921d4c241b8ce06 => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:03.929874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a7a28a-7f3d-468f-be16-6f4ea3f63fd5 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:03.930336 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a7a28a-7f3d-468f-be16-6f4ea3f63fd5 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:03.930519 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a7a28a-7f3d-468f-be16-6f4ea3f63fd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:03.953077 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-a2a7a28a-7f3d-468f-be16-6f4ea3f63fd5 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:03.954561 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a2a7a28a-7f3d-468f-be16-6f4ea3f63fd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:03.962760 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 980/1960] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:03.963537 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a7a28a-7f3d-468f-be16-6f4ea3f63fd5 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:03.965318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a2a7a28a-7f3d-468f-be16-6f4ea3f63fd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:03.977949 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a7a28a-7f3d-468f-be16-6f4ea3f63fd5 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:03.983164 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5949593e-3dba-40e6-a188-f0eadd04666f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:03.983910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5949593e-3dba-40e6-a188-f0eadd04666f admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:03.986247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5949593e-3dba-40e6-a188-f0eadd04666f admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:03.986247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5949593e-3dba-40e6-a188-f0eadd04666f admin admin] PATH_INFO: `/v3/projects/aa4000b6511c476eb95622a6c86f4cf7/users/3372a4db7fae497cb31be248aad12329/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:03.999201 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5949593e-3dba-40e6-a188-f0eadd04666f admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa4000b6511c476eb95622a6c86f4cf7, user_id=3372a4db7fae497cb31be248aad12329, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:04.001867 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5949593e-3dba-40e6-a188-f0eadd04666f admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:04.067777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 981/1961] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:03 2022] PUT /identity/v3/projects/aa4000b6511c476eb95622a6c86f4cf7/users/3372a4db7fae497cb31be248aad12329/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:04.068795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5949593e-3dba-40e6-a188-f0eadd04666f admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:04.070366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-5949593e-3dba-40e6-a188-f0eadd04666f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:04.101144 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-328a7821-cb19-4cf1-a802-c7854b336d51 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7eb49e88af3e422581776a81236f31af'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:04.104247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-328a7821-cb19-4cf1-a802-c7854b336d51 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7eb49e88af3e422581776a81236f31af'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:04.104247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-328a7821-cb19-4cf1-a802-c7854b336d51 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7eb49e88af3e422581776a81236f31af'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:04.109661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-328a7821-cb19-4cf1-a802-c7854b336d51 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7eb49e88af3e422581776a81236f31af'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:04.133094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5949593e-3dba-40e6-a188-f0eadd04666f admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:04.137793 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be3dae1-e37e-4592-9d4d-bbb596f06152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:04.138965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be3dae1-e37e-4592-9d4d-bbb596f06152 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:04.139239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be3dae1-e37e-4592-9d4d-bbb596f06152 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:04.139511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be3dae1-e37e-4592-9d4d-bbb596f06152 admin admin] PATH_INFO: `/v3/users/106c3ba56cd84155bedc474fd2479355` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:04.140652 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be3dae1-e37e-4592-9d4d-bbb596f06152 admin admin] RBAC: Authorizing `identity:delete_user(user_id=106c3ba56cd84155bedc474fd2479355)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:04.142575 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be3dae1-e37e-4592-9d4d-bbb596f06152 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:04.197436 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-328a7821-cb19-4cf1-a802-c7854b336d51 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7eb49e88af3e422581776a81236f31af'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:04.213439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 981/1962] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:03 2022] DELETE /identity/v3/users/7eb49e88af3e422581776a81236f31af => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:04.214816 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328a7821-cb19-4cf1-a802-c7854b336d51 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:04.215798 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-328a7821-cb19-4cf1-a802-c7854b336d51 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:04.234058 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:04.269037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:04.284833 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328a7821-cb19-4cf1-a802-c7854b336d51 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:04.289706 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b03a2d9-fe7d-4bb2-aaad-9af08681018d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:04.290339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b03a2d9-fe7d-4bb2-aaad-9af08681018d admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:04.290951 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b03a2d9-fe7d-4bb2-aaad-9af08681018d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:04.290951 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b03a2d9-fe7d-4bb2-aaad-9af08681018d admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:04.291653 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b03a2d9-fe7d-4bb2-aaad-9af08681018d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:47:04.291988 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b03a2d9-fe7d-4bb2-aaad-9af08681018d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:04.293645 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b03a2d9-fe7d-4bb2-aaad-9af08681018d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:04.305544 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 982/1963] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:47:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:47:04.310660 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca13dede-5359-491e-be96-c8feb28df2e9 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:04.310947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca13dede-5359-491e-be96-c8feb28df2e9 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:04.311223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca13dede-5359-491e-be96-c8feb28df2e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:04.339034 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ca13dede-5359-491e-be96-c8feb28df2e9 None None] MFA Rules not processed for user `3372a4db7fae497cb31be248aad12329`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:04.340248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ca13dede-5359-491e-be96-c8feb28df2e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:04.357085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8be3dae1-e37e-4592-9d4d-bbb596f06152 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '106c3ba56cd84155bedc474fd2479355'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:04.369143 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8be3dae1-e37e-4592-9d4d-bbb596f06152 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '106c3ba56cd84155bedc474fd2479355'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:04.370074 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8be3dae1-e37e-4592-9d4d-bbb596f06152 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '106c3ba56cd84155bedc474fd2479355'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:04.404173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 983/1964] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:04 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:04.405018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca13dede-5359-491e-be96-c8feb28df2e9 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:04.406294 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ca13dede-5359-491e-be96-c8feb28df2e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:04.419520 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca13dede-5359-491e-be96-c8feb28df2e9 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:04.424244 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3926ce-7178-46a4-8134-e98ad7cbf5a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:04.424879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3926ce-7178-46a4-8134-e98ad7cbf5a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:04.425161 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3926ce-7178-46a4-8134-e98ad7cbf5a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:04.425827 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3926ce-7178-46a4-8134-e98ad7cbf5a5 admin admin] PATH_INFO: `/v3/projects/fc07a91eb5904acaa15d88b72883af30` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:04.426562 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3926ce-7178-46a4-8134-e98ad7cbf5a5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fc07a91eb5904acaa15d88b72883af30)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:04.432887 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3926ce-7178-46a4-8134-e98ad7cbf5a5 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:04.460753 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8be3dae1-e37e-4592-9d4d-bbb596f06152 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '106c3ba56cd84155bedc474fd2479355'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:04.461400 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8be3dae1-e37e-4592-9d4d-bbb596f06152 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '106c3ba56cd84155bedc474fd2479355'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:04.467969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 982/1965] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:04 2022] DELETE /identity/v3/users/106c3ba56cd84155bedc474fd2479355 => generated 0 bytes in 400 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:04.468960 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be3dae1-e37e-4592-9d4d-bbb596f06152 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:04.470263 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-8be3dae1-e37e-4592-9d4d-bbb596f06152 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:04.550557 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be3dae1-e37e-4592-9d4d-bbb596f06152 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:04.555665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bc0c6f-5270-4edd-a715-a1dca5ddd1fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:04.556330 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bc0c6f-5270-4edd-a715-a1dca5ddd1fb admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:04.556608 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bc0c6f-5270-4edd-a715-a1dca5ddd1fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:04.556908 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bc0c6f-5270-4edd-a715-a1dca5ddd1fb admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:04.557569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bc0c6f-5270-4edd-a715-a1dca5ddd1fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:04.559680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bc0c6f-5270-4edd-a715-a1dca5ddd1fb admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:04.579716 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9f3926ce-7178-46a4-8134-e98ad7cbf5a5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fc07a91eb5904acaa15d88b72883af30', 'request_id': 'req-9f3926ce-7178-46a4-8134-e98ad7cbf5a5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:04.585061 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9f3926ce-7178-46a4-8134-e98ad7cbf5a5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fc07a91eb5904acaa15d88b72883af30', 'request_id': 'req-9f3926ce-7178-46a4-8134-e98ad7cbf5a5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:04.591051 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9f3926ce-7178-46a4-8134-e98ad7cbf5a5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fc07a91eb5904acaa15d88b72883af30', 'request_id': 'req-9f3926ce-7178-46a4-8134-e98ad7cbf5a5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:04.722101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9f3926ce-7178-46a4-8134-e98ad7cbf5a5 admin admin] The token cache is being invalidate because project fc07a91eb5904acaa15d88b72883af30 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:04.722539 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9f3926ce-7178-46a4-8134-e98ad7cbf5a5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:04.723761 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 983/1966] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:04 2022] POST /identity/v3/projects => generated 383 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:04.724662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bc0c6f-5270-4edd-a715-a1dca5ddd1fb admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:04.726966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9f3926ce-7178-46a4-8134-e98ad7cbf5a5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:04.728570 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 984/1967] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:04 2022] DELETE /identity/v3/projects/fc07a91eb5904acaa15d88b72883af30 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:04.729408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3926ce-7178-46a4-8134-e98ad7cbf5a5 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:04.730783 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9f3926ce-7178-46a4-8134-e98ad7cbf5a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:04.732827 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-00bc0c6f-5270-4edd-a715-a1dca5ddd1fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:04.823718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3926ce-7178-46a4-8134-e98ad7cbf5a5 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:04.831087 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805fe73d-4344-48cc-af2a-412237ed6367 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:04.832018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805fe73d-4344-48cc-af2a-412237ed6367 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:04.832394 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805fe73d-4344-48cc-af2a-412237ed6367 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:04.832788 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805fe73d-4344-48cc-af2a-412237ed6367 admin admin] PATH_INFO: `/v3/projects/48eb6d0c3f1f4cf98dd5707729fcf461` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:04.834370 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805fe73d-4344-48cc-af2a-412237ed6367 admin admin] RBAC: Authorizing `identity:delete_project(project_id=48eb6d0c3f1f4cf98dd5707729fcf461)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:04.836524 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805fe73d-4344-48cc-af2a-412237ed6367 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:04.837181 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bc0c6f-5270-4edd-a715-a1dca5ddd1fb admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:04.842763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99982393-0d9d-42b1-a976-2e22651e9d94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:04.843443 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99982393-0d9d-42b1-a976-2e22651e9d94 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:04.843789 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99982393-0d9d-42b1-a976-2e22651e9d94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:04.844131 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99982393-0d9d-42b1-a976-2e22651e9d94 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:04.844831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99982393-0d9d-42b1-a976-2e22651e9d94 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:04.846823 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99982393-0d9d-42b1-a976-2e22651e9d94 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:04.938374 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 984/1968] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:04 2022] POST /identity/v3/projects => generated 399 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:04.939375 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99982393-0d9d-42b1-a976-2e22651e9d94 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:04.940932 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-99982393-0d9d-42b1-a976-2e22651e9d94 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:04.981137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:04.997028 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-805fe73d-4344-48cc-af2a-412237ed6367 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '48eb6d0c3f1f4cf98dd5707729fcf461', 'request_id': 'req-805fe73d-4344-48cc-af2a-412237ed6367'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:05.006845 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-805fe73d-4344-48cc-af2a-412237ed6367 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '48eb6d0c3f1f4cf98dd5707729fcf461', 'request_id': 'req-805fe73d-4344-48cc-af2a-412237ed6367'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:05.010006 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99982393-0d9d-42b1-a976-2e22651e9d94 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:05.014282 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-805fe73d-4344-48cc-af2a-412237ed6367 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '48eb6d0c3f1f4cf98dd5707729fcf461', 'request_id': 'req-805fe73d-4344-48cc-af2a-412237ed6367'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:05.067564 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57f85bc-5a69-4e5e-8e80-3e1ac9563d34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:05.068181 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57f85bc-5a69-4e5e-8e80-3e1ac9563d34 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:05.068464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57f85bc-5a69-4e5e-8e80-3e1ac9563d34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:05.068737 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57f85bc-5a69-4e5e-8e80-3e1ac9563d34 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:05.069396 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57f85bc-5a69-4e5e-8e80-3e1ac9563d34 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:05.071517 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57f85bc-5a69-4e5e-8e80-3e1ac9563d34 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:05.096488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-805fe73d-4344-48cc-af2a-412237ed6367 admin admin] The token cache is being invalidate because project 48eb6d0c3f1f4cf98dd5707729fcf461 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:05.099740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-805fe73d-4344-48cc-af2a-412237ed6367 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:05.099740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-805fe73d-4344-48cc-af2a-412237ed6367 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:05.099740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 985/1969] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:04 2022] DELETE /identity/v3/projects/48eb6d0c3f1f4cf98dd5707729fcf461 => generated 0 bytes in 370 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:05.100393 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e52b7a-e11f-4866-88cf-2b2d2dfb4242 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:05.100670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e52b7a-e11f-4866-88cf-2b2d2dfb4242 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:05.100939 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e52b7a-e11f-4866-88cf-2b2d2dfb4242 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:05.120769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-68e52b7a-e11f-4866-88cf-2b2d2dfb4242 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:05.121947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-68e52b7a-e11f-4866-88cf-2b2d2dfb4242 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:05.129680 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 986/1970] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:05.131357 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e52b7a-e11f-4866-88cf-2b2d2dfb4242 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:05.132626 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-68e52b7a-e11f-4866-88cf-2b2d2dfb4242 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:05.144363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e52b7a-e11f-4866-88cf-2b2d2dfb4242 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:05.149047 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ca06af-570c-4866-b582-214af57232d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:05.149626 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ca06af-570c-4866-b582-214af57232d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:05.149906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ca06af-570c-4866-b582-214af57232d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:05.150181 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ca06af-570c-4866-b582-214af57232d2 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:05.150864 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ca06af-570c-4866-b582-214af57232d2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:05.152828 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ca06af-570c-4866-b582-214af57232d2 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:05.167951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 985/1971] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:04 2022] POST /identity/v3/users => generated 292 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:05.167951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57f85bc-5a69-4e5e-8e80-3e1ac9563d34 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:05.167951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b57f85bc-5a69-4e5e-8e80-3e1ac9563d34 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:05.171511 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:05.179642 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57f85bc-5a69-4e5e-8e80-3e1ac9563d34 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:05.184277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b69478d-40ea-40df-8660-c77fd482380b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:05.184897 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b69478d-40ea-40df-8660-c77fd482380b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:05.185178 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b69478d-40ea-40df-8660-c77fd482380b admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:05.185451 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b69478d-40ea-40df-8660-c77fd482380b admin admin] PATH_INFO: `/v3/projects/32153e84064043b8aaff202896b8d86c` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:05.186582 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b69478d-40ea-40df-8660-c77fd482380b admin admin] RBAC: Authorizing `identity:delete_project(project_id=32153e84064043b8aaff202896b8d86c)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:05.191209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b69478d-40ea-40df-8660-c77fd482380b admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:05.256837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 987/1972] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:05 2022] POST /identity/v3/users => generated 306 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:05.256837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d491d1-3ebe-43e2-9c13-b9e03ad08d26 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:05.256837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d491d1-3ebe-43e2-9c13-b9e03ad08d26 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:05.256837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d491d1-3ebe-43e2-9c13-b9e03ad08d26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:05.274805 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e4d491d1-3ebe-43e2-9c13-b9e03ad08d26 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:05.275994 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e4d491d1-3ebe-43e2-9c13-b9e03ad08d26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:05.284505 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 988/1973] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:05.285896 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f9d4d6-2c03-49e6-8fc3-c4b3b50827a2 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:05.286172 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f9d4d6-2c03-49e6-8fc3-c4b3b50827a2 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:05.286439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f9d4d6-2c03-49e6-8fc3-c4b3b50827a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:05.306244 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-74f9d4d6-2c03-49e6-8fc3-c4b3b50827a2 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:05.307478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-74f9d4d6-2c03-49e6-8fc3-c4b3b50827a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:05.315351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 989/1974] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:05.316283 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f9d4d6-2c03-49e6-8fc3-c4b3b50827a2 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:05.317581 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-74f9d4d6-2c03-49e6-8fc3-c4b3b50827a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:05.329532 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f9d4d6-2c03-49e6-8fc3-c4b3b50827a2 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:05.343567 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5b69478d-40ea-40df-8660-c77fd482380b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '32153e84064043b8aaff202896b8d86c', 'request_id': 'req-5b69478d-40ea-40df-8660-c77fd482380b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:05.349226 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5b69478d-40ea-40df-8660-c77fd482380b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '32153e84064043b8aaff202896b8d86c', 'request_id': 'req-5b69478d-40ea-40df-8660-c77fd482380b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:05.355998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5b69478d-40ea-40df-8660-c77fd482380b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '32153e84064043b8aaff202896b8d86c', 'request_id': 'req-5b69478d-40ea-40df-8660-c77fd482380b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:05.389384 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b38edc1-2d6f-4d44-9c92-710f2284b95b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:05.389988 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b38edc1-2d6f-4d44-9c92-710f2284b95b admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:05.390268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b38edc1-2d6f-4d44-9c92-710f2284b95b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:05.390542 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b38edc1-2d6f-4d44-9c92-710f2284b95b admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:05.391206 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b38edc1-2d6f-4d44-9c92-710f2284b95b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:05.392791 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b38edc1-2d6f-4d44-9c92-710f2284b95b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:05.402194 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 990/1975] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:05 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:05.403031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b38edc1-2d6f-4d44-9c92-710f2284b95b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:05.412806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b38edc1-2d6f-4d44-9c92-710f2284b95b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:05.417415 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7a0cee-c980-44cb-8533-8d2821281cc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:05.417978 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7a0cee-c980-44cb-8533-8d2821281cc3 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:05.418257 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7a0cee-c980-44cb-8533-8d2821281cc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:05.418530 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7a0cee-c980-44cb-8533-8d2821281cc3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:05.419173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7a0cee-c980-44cb-8533-8d2821281cc3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:05.420713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7a0cee-c980-44cb-8533-8d2821281cc3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:05.430154 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 991/1976] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:05 2022] GET /identity/v3/roles => generated 1420 bytes in 27 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:05.431303 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401681fc-79c8-458f-bc56-82d80592682f None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:05.431613 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401681fc-79c8-458f-bc56-82d80592682f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:05.431887 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401681fc-79c8-458f-bc56-82d80592682f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:05.450143 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5b69478d-40ea-40df-8660-c77fd482380b admin admin] The token cache is being invalidate because project 32153e84064043b8aaff202896b8d86c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:05.454663 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-401681fc-79c8-458f-bc56-82d80592682f None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:05.454797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5b69478d-40ea-40df-8660-c77fd482380b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:05.454797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5b69478d-40ea-40df-8660-c77fd482380b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:05.454967 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-401681fc-79c8-458f-bc56-82d80592682f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:05.455082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 986/1977] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:05 2022] DELETE /identity/v3/projects/32153e84064043b8aaff202896b8d86c => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:05.455082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b69478d-40ea-40df-8660-c77fd482380b admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:05.457570 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-5b69478d-40ea-40df-8660-c77fd482380b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:05.460593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 992/1978] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:05.461232 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401681fc-79c8-458f-bc56-82d80592682f None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:05.464573 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-401681fc-79c8-458f-bc56-82d80592682f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:05.473444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b69478d-40ea-40df-8660-c77fd482380b admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:05.477121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401681fc-79c8-458f-bc56-82d80592682f None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:05.478102 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855fba8b-756f-4021-b67a-af1fa737c63f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:05.478737 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855fba8b-756f-4021-b67a-af1fa737c63f admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:05.479021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855fba8b-756f-4021-b67a-af1fa737c63f admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:05.479296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855fba8b-756f-4021-b67a-af1fa737c63f admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:05.479944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855fba8b-756f-4021-b67a-af1fa737c63f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:47:05.480247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855fba8b-756f-4021-b67a-af1fa737c63f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:05.481874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855fba8b-756f-4021-b67a-af1fa737c63f admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:05.481986 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61d5cfe-9506-4b24-997b-a10d9619ec25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:05.482486 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61d5cfe-9506-4b24-997b-a10d9619ec25 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:05.482793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61d5cfe-9506-4b24-997b-a10d9619ec25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:05.483121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61d5cfe-9506-4b24-997b-a10d9619ec25 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:05.483765 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61d5cfe-9506-4b24-997b-a10d9619ec25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:05.485381 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61d5cfe-9506-4b24-997b-a10d9619ec25 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:05.493705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 987/1979] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:47:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:47:05.494489 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855fba8b-756f-4021-b67a-af1fa737c63f admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:05.495649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 993/1980] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:05 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:05.496834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e107df9b-0df1-468f-bb6c-1d757fd397e6 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:05.500489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e107df9b-0df1-468f-bb6c-1d757fd397e6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:05.500489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e107df9b-0df1-468f-bb6c-1d757fd397e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:05.500676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-855fba8b-756f-4021-b67a-af1fa737c63f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:05.511905 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855fba8b-756f-4021-b67a-af1fa737c63f admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:05.517189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff7da8f-89b2-4a22-834c-9e96a1943b95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:05.517854 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e107df9b-0df1-468f-bb6c-1d757fd397e6 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:05.517984 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff7da8f-89b2-4a22-834c-9e96a1943b95 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:05.518269 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff7da8f-89b2-4a22-834c-9e96a1943b95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:05.518548 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff7da8f-89b2-4a22-834c-9e96a1943b95 admin admin] PATH_INFO: `/v3/projects/014647dd60d54c4791f979e379e6be42/users/dfbe496c221648908e1cc158ef6f10c5/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:05.523394 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e107df9b-0df1-468f-bb6c-1d757fd397e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:05.527131 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 994/1981] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:05.527908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e107df9b-0df1-468f-bb6c-1d757fd397e6 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:05.534446 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff7da8f-89b2-4a22-834c-9e96a1943b95 admin admin] RBAC: Authorizing `identity:create_grant(project_id=014647dd60d54c4791f979e379e6be42, user_id=dfbe496c221648908e1cc158ef6f10c5, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:05.537398 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff7da8f-89b2-4a22-834c-9e96a1943b95 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:05.537969 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e107df9b-0df1-468f-bb6c-1d757fd397e6 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:05.542935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30daff3a-527a-4023-9e58-6a3881483158 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:05.543532 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30daff3a-527a-4023-9e58-6a3881483158 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:05.543825 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30daff3a-527a-4023-9e58-6a3881483158 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:05.544075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30daff3a-527a-4023-9e58-6a3881483158 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:05.544786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30daff3a-527a-4023-9e58-6a3881483158 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:47:05.545082 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30daff3a-527a-4023-9e58-6a3881483158 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:05.546749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30daff3a-527a-4023-9e58-6a3881483158 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:05.558225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 995/1982] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:47:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:47:05.559001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30daff3a-527a-4023-9e58-6a3881483158 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:05.568269 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30daff3a-527a-4023-9e58-6a3881483158 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:05.573169 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1e1415-89d7-4570-ae80-0aff729f70a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:05.573738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1e1415-89d7-4570-ae80-0aff729f70a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:05.574041 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1e1415-89d7-4570-ae80-0aff729f70a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:05.574340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1e1415-89d7-4570-ae80-0aff729f70a5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:05.574997 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1e1415-89d7-4570-ae80-0aff729f70a5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:05.577429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1e1415-89d7-4570-ae80-0aff729f70a5 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:05.629637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 988/1983] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:05 2022] PUT /identity/v3/projects/014647dd60d54c4791f979e379e6be42/users/dfbe496c221648908e1cc158ef6f10c5/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:05.630637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff7da8f-89b2-4a22-834c-9e96a1943b95 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:05.671286 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 996/1984] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:05 2022] POST /identity/v3/projects => generated 375 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:05.672528 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01d9b4b-1608-4b47-96a4-a111dce65c62 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:05.672815 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01d9b4b-1608-4b47-96a4-a111dce65c62 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:05.673083 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01d9b4b-1608-4b47-96a4-a111dce65c62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:05.691581 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c01d9b4b-1608-4b47-96a4-a111dce65c62 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:05.693676 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c01d9b4b-1608-4b47-96a4-a111dce65c62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:05.694757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff7da8f-89b2-4a22-834c-9e96a1943b95 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:05.733417 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:05.751506 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 997/1985] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:05.751657 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c938885e-3048-41b3-8e12-b68ab03795a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:05.752248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01d9b4b-1608-4b47-96a4-a111dce65c62 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:05.752426 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c938885e-3048-41b3-8e12-b68ab03795a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:05.752857 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c938885e-3048-41b3-8e12-b68ab03795a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:05.753201 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c938885e-3048-41b3-8e12-b68ab03795a5 admin admin] PATH_INFO: `/v3/projects/82de32938a7340b391a698ddf94c8390/users/2087cc96a15646528ee3c5d14443bef6/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:05.758874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:05.762518 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01d9b4b-1608-4b47-96a4-a111dce65c62 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:05.767675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b52799-4e1d-48a3-9a31-90fa3c163f68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:05.768294 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b52799-4e1d-48a3-9a31-90fa3c163f68 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:05.768600 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b52799-4e1d-48a3-9a31-90fa3c163f68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:05.768786 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c938885e-3048-41b3-8e12-b68ab03795a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=82de32938a7340b391a698ddf94c8390, user_id=2087cc96a15646528ee3c5d14443bef6, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:05.768912 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b52799-4e1d-48a3-9a31-90fa3c163f68 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:05.769497 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b52799-4e1d-48a3-9a31-90fa3c163f68 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:05.771397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b52799-4e1d-48a3-9a31-90fa3c163f68 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:05.771612 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c938885e-3048-41b3-8e12-b68ab03795a5 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:05.846956 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 989/1986] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:05 2022] PUT /identity/v3/projects/82de32938a7340b391a698ddf94c8390/users/2087cc96a15646528ee3c5d14443bef6/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:05.847816 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c938885e-3048-41b3-8e12-b68ab03795a5 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:05.879220 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 998/1987] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:05 2022] POST /identity/v3/projects => generated 385 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:05.881001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b52799-4e1d-48a3-9a31-90fa3c163f68 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:05.931434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c938885e-3048-41b3-8e12-b68ab03795a5 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:05.960889 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b52799-4e1d-48a3-9a31-90fa3c163f68 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:05.967007 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f5a013-1268-4c6c-a993-bda4b2795989 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:05.967629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f5a013-1268-4c6c-a993-bda4b2795989 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:05.968045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f5a013-1268-4c6c-a993-bda4b2795989 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:05.968332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f5a013-1268-4c6c-a993-bda4b2795989 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:05.968974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f5a013-1268-4c6c-a993-bda4b2795989 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:05.971271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f5a013-1268-4c6c-a993-bda4b2795989 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:05.992045 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5d80ed-225d-412e-ae5f-dab9635011e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:05.992657 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5d80ed-225d-412e-ae5f-dab9635011e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:05.992930 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5d80ed-225d-412e-ae5f-dab9635011e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:05.993232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5d80ed-225d-412e-ae5f-dab9635011e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:05.993871 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5d80ed-225d-412e-ae5f-dab9635011e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:05.995752 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5d80ed-225d-412e-ae5f-dab9635011e2 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:06.006636 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 990/1988] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:05 2022] GET /identity/v3/roles => generated 1420 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:06.010796 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5d80ed-225d-412e-ae5f-dab9635011e2 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:06.019198 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5d80ed-225d-412e-ae5f-dab9635011e2 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:06.024009 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d224e627-3e8d-423f-aff1-6d0a515b4507 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:06.024586 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d224e627-3e8d-423f-aff1-6d0a515b4507 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:06.024862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d224e627-3e8d-423f-aff1-6d0a515b4507 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:06.025128 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d224e627-3e8d-423f-aff1-6d0a515b4507 admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:06.025754 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d224e627-3e8d-423f-aff1-6d0a515b4507 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:47:06.026053 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d224e627-3e8d-423f-aff1-6d0a515b4507 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:06.027693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d224e627-3e8d-423f-aff1-6d0a515b4507 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:06.039412 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 991/1989] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:47:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:47:06.040529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21353a59-6fea-43ae-886a-1d524132ec7a None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:06.040812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21353a59-6fea-43ae-886a-1d524132ec7a None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:06.041079 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21353a59-6fea-43ae-886a-1d524132ec7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:06.064480 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 999/1990] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:05 2022] POST /identity/v3/users => generated 288 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:06.065401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f5a013-1268-4c6c-a993-bda4b2795989 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:06.069608 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-21353a59-6fea-43ae-886a-1d524132ec7a None None] MFA Rules not processed for user `2087cc96a15646528ee3c5d14443bef6`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:06.070883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-21353a59-6fea-43ae-886a-1d524132ec7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:06.075290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f5a013-1268-4c6c-a993-bda4b2795989 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:06.081857 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:DELETE=5,keystone:SELECT=92,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:06.082004 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51216840-8e1f-404c-a776-251ffac6ccde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:06.082606 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51216840-8e1f-404c-a776-251ffac6ccde admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:06.082897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51216840-8e1f-404c-a776-251ffac6ccde admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:06.083171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51216840-8e1f-404c-a776-251ffac6ccde admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:06.083793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51216840-8e1f-404c-a776-251ffac6ccde admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:06.085740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51216840-8e1f-404c-a776-251ffac6ccde admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:06.136628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 992/1991] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:06 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:06.140843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21353a59-6fea-43ae-886a-1d524132ec7a None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:06.147873 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21353a59-6fea-43ae-886a-1d524132ec7a None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:06.152566 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30467ff-9c62-46a0-870d-df6607bad08f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:06.153222 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30467ff-9c62-46a0-870d-df6607bad08f admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:06.153479 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30467ff-9c62-46a0-870d-df6607bad08f admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:06.153776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30467ff-9c62-46a0-870d-df6607bad08f admin admin] PATH_INFO: `/v3/projects/014647dd60d54c4791f979e379e6be42/users/dfbe496c221648908e1cc158ef6f10c5/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:06.155685 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30467ff-9c62-46a0-870d-df6607bad08f admin admin] RBAC: Authorizing `identity:create_grant(project_id=014647dd60d54c4791f979e379e6be42, user_id=dfbe496c221648908e1cc158ef6f10c5, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:06.158374 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30467ff-9c62-46a0-870d-df6607bad08f admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:06.168239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 993/1992] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:06 2022] PUT /identity/v3/projects/014647dd60d54c4791f979e379e6be42/users/dfbe496c221648908e1cc158ef6f10c5/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:06.168994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30467ff-9c62-46a0-870d-df6607bad08f admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:06.178329 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:06.189245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1000/1993] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:06 2022] POST /identity/v3/users => generated 293 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:06.189245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51216840-8e1f-404c-a776-251ffac6ccde admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:06.197636 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51216840-8e1f-404c-a776-251ffac6ccde admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:06.235344 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30467ff-9c62-46a0-870d-df6607bad08f admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:06.240015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea84f098-5447-4985-b7b0-252a96184595 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:06.240652 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea84f098-5447-4985-b7b0-252a96184595 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:06.240938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea84f098-5447-4985-b7b0-252a96184595 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:06.241213 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea84f098-5447-4985-b7b0-252a96184595 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:06.241862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea84f098-5447-4985-b7b0-252a96184595 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:06.243821 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea84f098-5447-4985-b7b0-252a96184595 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:06.257120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5707226-e93f-4a57-913c-4efd5ce4aa4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:06.257678 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5707226-e93f-4a57-913c-4efd5ce4aa4d admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:06.257935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5707226-e93f-4a57-913c-4efd5ce4aa4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:06.258205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5707226-e93f-4a57-913c-4efd5ce4aa4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:06.260539 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5707226-e93f-4a57-913c-4efd5ce4aa4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:06.260891 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5707226-e93f-4a57-913c-4efd5ce4aa4d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:06.271257 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1001/1994] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:06 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:06.271257 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5707226-e93f-4a57-913c-4efd5ce4aa4d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:06.282235 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5707226-e93f-4a57-913c-4efd5ce4aa4d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:06.287086 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803faf25-1b4c-4fb8-b4b9-179d6f85213c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:06.287674 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803faf25-1b4c-4fb8-b4b9-179d6f85213c admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:06.288165 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803faf25-1b4c-4fb8-b4b9-179d6f85213c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:06.288301 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803faf25-1b4c-4fb8-b4b9-179d6f85213c admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:06.288875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803faf25-1b4c-4fb8-b4b9-179d6f85213c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:06.290792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803faf25-1b4c-4fb8-b4b9-179d6f85213c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:06.337426 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 994/1995] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:06 2022] POST /identity/v3/projects => generated 363 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:06.338890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4964cabe-ebc7-40ea-84ef-d36104212006 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:06.339252 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4964cabe-ebc7-40ea-84ef-d36104212006 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:06.339601 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4964cabe-ebc7-40ea-84ef-d36104212006 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:06.367121 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-4964cabe-ebc7-40ea-84ef-d36104212006 None None] MFA Rules not processed for user `dfbe496c221648908e1cc158ef6f10c5`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:06.368349 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-4964cabe-ebc7-40ea-84ef-d36104212006 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:06.408448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1002/1996] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:06 2022] POST /identity/v3/projects => generated 383 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:06.409392 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803faf25-1b4c-4fb8-b4b9-179d6f85213c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:06.438968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 995/1997] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:06 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:06.439791 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4964cabe-ebc7-40ea-84ef-d36104212006 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:06.467289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:06.472112 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803faf25-1b4c-4fb8-b4b9-179d6f85213c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:06.476775 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa526c1-8aba-434d-b9c1-e389779e2454 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:06.477388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa526c1-8aba-434d-b9c1-e389779e2454 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:06.477666 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa526c1-8aba-434d-b9c1-e389779e2454 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:06.477936 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa526c1-8aba-434d-b9c1-e389779e2454 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:06.478571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa526c1-8aba-434d-b9c1-e389779e2454 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:06.480227 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa526c1-8aba-434d-b9c1-e389779e2454 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:06.490215 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1003/1998] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:06 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:06.491551 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa526c1-8aba-434d-b9c1-e389779e2454 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:06.499924 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:06.503679 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa526c1-8aba-434d-b9c1-e389779e2454 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:06.504312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4964cabe-ebc7-40ea-84ef-d36104212006 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:06.508455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8b9cbd-14bb-4e56-ba67-dbab26c115d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:06.508993 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8b9cbd-14bb-4e56-ba67-dbab26c115d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:06.509209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0256c8-4d7e-4d68-bcf6-73c587b6cbfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:06.509322 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8b9cbd-14bb-4e56-ba67-dbab26c115d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:06.509523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8b9cbd-14bb-4e56-ba67-dbab26c115d6 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:06.509867 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0256c8-4d7e-4d68-bcf6-73c587b6cbfd admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:06.510140 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8b9cbd-14bb-4e56-ba67-dbab26c115d6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:06.510244 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0256c8-4d7e-4d68-bcf6-73c587b6cbfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:06.510411 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0256c8-4d7e-4d68-bcf6-73c587b6cbfd admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:06.511081 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0256c8-4d7e-4d68-bcf6-73c587b6cbfd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:06.512111 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8b9cbd-14bb-4e56-ba67-dbab26c115d6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:06.512701 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0256c8-4d7e-4d68-bcf6-73c587b6cbfd admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:06.523628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 996/1999] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:06 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:06.524438 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0256c8-4d7e-4d68-bcf6-73c587b6cbfd admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:06.534458 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0256c8-4d7e-4d68-bcf6-73c587b6cbfd admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:06.539334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8255b11b-e0b6-4fd7-a22b-c6499ced44d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:06.539901 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8255b11b-e0b6-4fd7-a22b-c6499ced44d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:06.540180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8255b11b-e0b6-4fd7-a22b-c6499ced44d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:06.540452 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8255b11b-e0b6-4fd7-a22b-c6499ced44d0 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:06.541065 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8255b11b-e0b6-4fd7-a22b-c6499ced44d0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:06.543116 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8255b11b-e0b6-4fd7-a22b-c6499ced44d0 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:06.635556 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1004/2000] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:06 2022] POST /identity/v3/users => generated 282 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:06.636460 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8b9cbd-14bb-4e56-ba67-dbab26c115d6 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:06.638676 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8a8b9cbd-14bb-4e56-ba67-dbab26c115d6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:06.663810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 997/2001] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:06 2022] POST /identity/v3/users => generated 292 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:06.668982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8255b11b-e0b6-4fd7-a22b-c6499ced44d0 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:06.674746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8255b11b-e0b6-4fd7-a22b-c6499ced44d0 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:06.679424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af577e6d-0a2a-4eeb-90fc-3cb1bf3ae24b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:06.680010 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af577e6d-0a2a-4eeb-90fc-3cb1bf3ae24b admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:06.680295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af577e6d-0a2a-4eeb-90fc-3cb1bf3ae24b admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:06.680573 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af577e6d-0a2a-4eeb-90fc-3cb1bf3ae24b admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:06.681209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af577e6d-0a2a-4eeb-90fc-3cb1bf3ae24b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:06.682877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af577e6d-0a2a-4eeb-90fc-3cb1bf3ae24b admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:06.694175 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 998/2002] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:06 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:06.694175 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af577e6d-0a2a-4eeb-90fc-3cb1bf3ae24b admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:06.700022 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8b9cbd-14bb-4e56-ba67-dbab26c115d6 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:06.704502 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4df86c1-e2cd-4e58-9181-56628c502cd7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:06.705118 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4df86c1-e2cd-4e58-9181-56628c502cd7 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:06.705398 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4df86c1-e2cd-4e58-9181-56628c502cd7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:06.707415 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4df86c1-e2cd-4e58-9181-56628c502cd7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:06.707517 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af577e6d-0a2a-4eeb-90fc-3cb1bf3ae24b admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:06.711658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b033ce-4760-4d5e-89e7-e60d29460b43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:06.712294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b033ce-4760-4d5e-89e7-e60d29460b43 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:06.712575 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b033ce-4760-4d5e-89e7-e60d29460b43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:06.712864 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b033ce-4760-4d5e-89e7-e60d29460b43 admin admin] PATH_INFO: `/v3/projects/b39a7111b78845219da0f1c53007fc4a/users/a0ddd0819da6453fb289ac068bdde657/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:06.727753 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b033ce-4760-4d5e-89e7-e60d29460b43 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b39a7111b78845219da0f1c53007fc4a, user_id=a0ddd0819da6453fb289ac068bdde657, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:06.730450 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b033ce-4760-4d5e-89e7-e60d29460b43 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:06.767369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4df86c1-e2cd-4e58-9181-56628c502cd7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:06.768737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4df86c1-e2cd-4e58-9181-56628c502cd7 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:06.778150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1005/2003] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:06 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:06.779048 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4df86c1-e2cd-4e58-9181-56628c502cd7 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:06.788929 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4df86c1-e2cd-4e58-9181-56628c502cd7 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:06.794188 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adfd5915-a015-4054-b2a4-51aa62564009 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:06.799007 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfd5915-a015-4054-b2a4-51aa62564009 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:06.799007 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfd5915-a015-4054-b2a4-51aa62564009 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:06.799007 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfd5915-a015-4054-b2a4-51aa62564009 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:06.799007 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adfd5915-a015-4054-b2a4-51aa62564009 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:06.799007 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adfd5915-a015-4054-b2a4-51aa62564009 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:06.807521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1006/2004] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:06 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:06.808335 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adfd5915-a015-4054-b2a4-51aa62564009 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:06.818529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adfd5915-a015-4054-b2a4-51aa62564009 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:06.823748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 999/2005] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:06 2022] PUT /identity/v3/projects/b39a7111b78845219da0f1c53007fc4a/users/a0ddd0819da6453fb289ac068bdde657/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:06.824381 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b033ce-4760-4d5e-89e7-e60d29460b43 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:06.842840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:06.885174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a71e02-eda2-4882-9a0d-acb6e7962c61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:06.885824 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a71e02-eda2-4882-9a0d-acb6e7962c61 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:06.886106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a71e02-eda2-4882-9a0d-acb6e7962c61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:06.886497 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a71e02-eda2-4882-9a0d-acb6e7962c61 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:06.887202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a71e02-eda2-4882-9a0d-acb6e7962c61 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:06.889408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a71e02-eda2-4882-9a0d-acb6e7962c61 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:06.895060 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b033ce-4760-4d5e-89e7-e60d29460b43 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:06.899319 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1007/2006] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:06 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:06.899827 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6265cb-e993-48fd-b133-e0f4506157bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:06.900363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a71e02-eda2-4882-9a0d-acb6e7962c61 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:06.900496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6265cb-e993-48fd-b133-e0f4506157bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:06.900768 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6265cb-e993-48fd-b133-e0f4506157bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:06.901057 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6265cb-e993-48fd-b133-e0f4506157bb admin admin] PATH_INFO: `/v3/projects/fcd06564ba0843d6b521120bebe94b95/users/bc75456a74bd456698580f490ad051f0/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:06.910403 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a71e02-eda2-4882-9a0d-acb6e7962c61 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:06.916476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6265cb-e993-48fd-b133-e0f4506157bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=fcd06564ba0843d6b521120bebe94b95, user_id=bc75456a74bd456698580f490ad051f0, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:06.917735 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4968e8c3-c482-4333-87d3-2cd41da3fc74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:06.918582 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4968e8c3-c482-4333-87d3-2cd41da3fc74 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:06.918904 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4968e8c3-c482-4333-87d3-2cd41da3fc74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:06.919179 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4968e8c3-c482-4333-87d3-2cd41da3fc74 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:06.920012 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6265cb-e993-48fd-b133-e0f4506157bb admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:06.920444 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4968e8c3-c482-4333-87d3-2cd41da3fc74 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:06.922163 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4968e8c3-c482-4333-87d3-2cd41da3fc74 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:06.937244 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1008/2007] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:06 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:06.937244 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47eb848-295c-480e-b7a0-78236c0eeae3 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:06.937244 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47eb848-295c-480e-b7a0-78236c0eeae3 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:06.942837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:06.942837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47eb848-295c-480e-b7a0-78236c0eeae3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:06.964651 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e47eb848-295c-480e-b7a0-78236c0eeae3 None None] MFA Rules not processed for user `a0ddd0819da6453fb289ac068bdde657`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:06.965861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e47eb848-295c-480e-b7a0-78236c0eeae3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:06.999166 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1000/2008] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:06 2022] PUT /identity/v3/projects/fcd06564ba0843d6b521120bebe94b95/users/bc75456a74bd456698580f490ad051f0/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:07.000940 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6265cb-e993-48fd-b133-e0f4506157bb admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:07.035624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1009/2009] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:06 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:07.036997 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47eb848-295c-480e-b7a0-78236c0eeae3 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:07.052500 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47eb848-295c-480e-b7a0-78236c0eeae3 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:07.081105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6265cb-e993-48fd-b133-e0f4506157bb admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:07.086261 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bfea13-d0c2-4b4a-806d-b72d682e765a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:07.086915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bfea13-d0c2-4b4a-806d-b72d682e765a admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:07.087202 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bfea13-d0c2-4b4a-806d-b72d682e765a admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:07.087481 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bfea13-d0c2-4b4a-806d-b72d682e765a admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:07.088136 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bfea13-d0c2-4b4a-806d-b72d682e765a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:07.089745 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bfea13-d0c2-4b4a-806d-b72d682e765a admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:07.100518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1001/2010] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:06 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:07.101891 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f8f347-fff5-4e2c-bfdb-a936c378edd3 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:07.102087 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f8f347-fff5-4e2c-bfdb-a936c378edd3 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:07.102547 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f8f347-fff5-4e2c-bfdb-a936c378edd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:07.123659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:07.128865 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea5adc8-a4f7-428c-ae23-5c46b01efd2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:07.129551 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea5adc8-a4f7-428c-ae23-5c46b01efd2f admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:07.129833 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea5adc8-a4f7-428c-ae23-5c46b01efd2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:07.130110 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea5adc8-a4f7-428c-ae23-5c46b01efd2f admin admin] PATH_INFO: `/v3/projects/e8f3f11ba4544c259c7fb8c6012443cb/users/4c7ee08a9a4848698ac7e1c21f08d5e0/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:07.137382 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-49f8f347-fff5-4e2c-bfdb-a936c378edd3 None None] MFA Rules not processed for user `bc75456a74bd456698580f490ad051f0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:07.138841 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-49f8f347-fff5-4e2c-bfdb-a936c378edd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:07.144967 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea5adc8-a4f7-428c-ae23-5c46b01efd2f admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8f3f11ba4544c259c7fb8c6012443cb, user_id=4c7ee08a9a4848698ac7e1c21f08d5e0, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:07.147734 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea5adc8-a4f7-428c-ae23-5c46b01efd2f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:07.155055 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:07.207917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1002/2011] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:07 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:07.208759 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f8f347-fff5-4e2c-bfdb-a936c378edd3 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:07.263747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1010/2012] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:07 2022] PUT /identity/v3/projects/e8f3f11ba4544c259c7fb8c6012443cb/users/4c7ee08a9a4848698ac7e1c21f08d5e0/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:07.264666 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea5adc8-a4f7-428c-ae23-5c46b01efd2f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:07.266236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-bea5adc8-a4f7-428c-ae23-5c46b01efd2f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:07.270676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f8f347-fff5-4e2c-bfdb-a936c378edd3 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:07.323434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d144919d-5769-495e-bc8d-254823072c69 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:07.324561 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d144919d-5769-495e-bc8d-254823072c69 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:07.324955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d144919d-5769-495e-bc8d-254823072c69 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:07.325283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d144919d-5769-495e-bc8d-254823072c69 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:07.327259 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d144919d-5769-495e-bc8d-254823072c69 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:07.347769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea5adc8-a4f7-428c-ae23-5c46b01efd2f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:07.351614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fa03d3-7e8c-46d9-949a-e36847cc4bdb service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf907d4033064fb8a2707b1bbb0a180b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:07.352299 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fa03d3-7e8c-46d9-949a-e36847cc4bdb service glance] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:07.352639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fa03d3-7e8c-46d9-949a-e36847cc4bdb service glance] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:07.353029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fa03d3-7e8c-46d9-949a-e36847cc4bdb service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:07.367892 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:07.394907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d144919d-5769-495e-bc8d-254823072c69 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:07.395285 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:07.396398 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d144919d-5769-495e-bc8d-254823072c69 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:07.406044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1003/2013] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:07 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:07.406933 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d144919d-5769-495e-bc8d-254823072c69 service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:07.413579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fa03d3-7e8c-46d9-949a-e36847cc4bdb service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:07.415113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fa03d3-7e8c-46d9-949a-e36847cc4bdb service glance] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:07.426833 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1011/2014] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:07 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:07.426833 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fa03d3-7e8c-46d9-949a-e36847cc4bdb service glance] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:07.484606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d144919d-5769-495e-bc8d-254823072c69 service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:07.489577 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb258b76-007b-4664-9a7f-62592bb090b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:07.490247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb258b76-007b-4664-9a7f-62592bb090b5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:07.500548 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:07.500750 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb258b76-007b-4664-9a7f-62592bb090b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:07.500750 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb258b76-007b-4664-9a7f-62592bb090b5 admin admin] PATH_INFO: `/v3/projects/a78c4b96aca44d8aa8cf6be70b245120/users/4d73b57c74c44615bc34b59348b6e1e7/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:07.503966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fa03d3-7e8c-46d9-949a-e36847cc4bdb service glance] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:07.509891 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb258b76-007b-4664-9a7f-62592bb090b5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a78c4b96aca44d8aa8cf6be70b245120, user_id=4d73b57c74c44615bc34b59348b6e1e7, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:07.514468 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb258b76-007b-4664-9a7f-62592bb090b5 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:07.517076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b596ce-8809-4480-a719-8bd08eb7d9a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:07.517768 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b596ce-8809-4480-a719-8bd08eb7d9a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:07.518052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b596ce-8809-4480-a719-8bd08eb7d9a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:07.518325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b596ce-8809-4480-a719-8bd08eb7d9a1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:07.519264 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b596ce-8809-4480-a719-8bd08eb7d9a1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:07.521300 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b596ce-8809-4480-a719-8bd08eb7d9a1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:07.526094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:07.624444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1004/2015] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:07 2022] PUT /identity/v3/projects/a78c4b96aca44d8aa8cf6be70b245120/users/4d73b57c74c44615bc34b59348b6e1e7/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:07.628851 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a701217-f9ac-4685-84cd-f3df14425cda None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:07.628851 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a701217-f9ac-4685-84cd-f3df14425cda None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:07.628851 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a701217-f9ac-4685-84cd-f3df14425cda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:07.658059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1012/2016] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:07 2022] POST /identity/v3/projects => generated 387 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:07.659840 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec3bb4e-4d4f-48c5-ad04-89f5d8347ecd None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:07.660141 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec3bb4e-4d4f-48c5-ad04-89f5d8347ecd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:07.660422 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec3bb4e-4d4f-48c5-ad04-89f5d8347ecd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:07.664680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-0a701217-f9ac-4685-84cd-f3df14425cda None None] MFA Rules not processed for user `4c7ee08a9a4848698ac7e1c21f08d5e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:07.665911 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0a701217-f9ac-4685-84cd-f3df14425cda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:07.697875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-5ec3bb4e-4d4f-48c5-ad04-89f5d8347ecd None None] MFA Rules not processed for user `4d73b57c74c44615bc34b59348b6e1e7`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:07.699248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5ec3bb4e-4d4f-48c5-ad04-89f5d8347ecd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:07.734921 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1005/2017] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:07 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:07.735746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a701217-f9ac-4685-84cd-f3df14425cda None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:07.780030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1013/2018] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:07 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:07.780930 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec3bb4e-4d4f-48c5-ad04-89f5d8347ecd None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:07.802448 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a701217-f9ac-4685-84cd-f3df14425cda None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:07.807619 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e5ac6c-26e1-4297-8745-f461eb4c350f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:07.812787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e5ac6c-26e1-4297-8745-f461eb4c350f admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:07.812787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e5ac6c-26e1-4297-8745-f461eb4c350f admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:07.812787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e5ac6c-26e1-4297-8745-f461eb4c350f admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:07.812787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e5ac6c-26e1-4297-8745-f461eb4c350f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:07.812787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e5ac6c-26e1-4297-8745-f461eb4c350f admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:07.847086 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec3bb4e-4d4f-48c5-ad04-89f5d8347ecd None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:07.851766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93952d8-834b-4af7-be0a-6634879ec7ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:07.852382 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93952d8-834b-4af7-be0a-6634879ec7ee admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:07.852660 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93952d8-834b-4af7-be0a-6634879ec7ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:07.852932 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93952d8-834b-4af7-be0a-6634879ec7ee admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:07.853566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93952d8-834b-4af7-be0a-6634879ec7ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:07.855622 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93952d8-834b-4af7-be0a-6634879ec7ee admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:07.858882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:07.919776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1006/2019] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:07 2022] POST /identity/v3/users => generated 294 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:07.921007 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e5ac6c-26e1-4297-8745-f461eb4c350f admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:07.930811 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e5ac6c-26e1-4297-8745-f461eb4c350f admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:07.936165 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17c211b-8695-45aa-86fd-3b23ebef2fa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:07.936740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17c211b-8695-45aa-86fd-3b23ebef2fa6 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:07.937020 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17c211b-8695-45aa-86fd-3b23ebef2fa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:07.937297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17c211b-8695-45aa-86fd-3b23ebef2fa6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:07.937913 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17c211b-8695-45aa-86fd-3b23ebef2fa6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:07.939893 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17c211b-8695-45aa-86fd-3b23ebef2fa6 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:07.951902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1014/2020] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:07 2022] POST /identity/v3/projects => generated 365 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:07.953044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93952d8-834b-4af7-be0a-6634879ec7ee admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:08.007355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:08.017549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93952d8-834b-4af7-be0a-6634879ec7ee admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:08.022243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89694e12-36a6-450e-907d-d1aab669f010 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:08.022864 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89694e12-36a6-450e-907d-d1aab669f010 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:08.023140 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89694e12-36a6-450e-907d-d1aab669f010 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:08.023414 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89694e12-36a6-450e-907d-d1aab669f010 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:08.024047 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89694e12-36a6-450e-907d-d1aab669f010 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:08.025630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89694e12-36a6-450e-907d-d1aab669f010 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:08.035889 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1015/2021] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:07 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:08.036472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89694e12-36a6-450e-907d-d1aab669f010 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:08.046349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89694e12-36a6-450e-907d-d1aab669f010 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:08.050959 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fba784-61d7-415b-b235-345af0fb9773 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:08.051527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fba784-61d7-415b-b235-345af0fb9773 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:08.051804 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fba784-61d7-415b-b235-345af0fb9773 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:08.052071 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fba784-61d7-415b-b235-345af0fb9773 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:08.052674 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99fba784-61d7-415b-b235-345af0fb9773 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:08.062823 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99fba784-61d7-415b-b235-345af0fb9773 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:08.098259 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1007/2022] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:07 2022] POST /identity/v3/projects => generated 385 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:08.098259 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17c211b-8695-45aa-86fd-3b23ebef2fa6 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:08.178997 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17c211b-8695-45aa-86fd-3b23ebef2fa6 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:08.182152 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1016/2023] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:08 2022] POST /identity/v3/users => generated 289 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:08.182152 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fba784-61d7-415b-b235-345af0fb9773 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:08.184406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df66fb7-11e3-470e-8e46-578d9d0ef705 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:08.185059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df66fb7-11e3-470e-8e46-578d9d0ef705 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:08.185338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df66fb7-11e3-470e-8e46-578d9d0ef705 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:08.185611 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df66fb7-11e3-470e-8e46-578d9d0ef705 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:08.186257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df66fb7-11e3-470e-8e46-578d9d0ef705 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:08.188036 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df66fb7-11e3-470e-8e46-578d9d0ef705 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:08.193427 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fba784-61d7-415b-b235-345af0fb9773 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:08.198200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6277b9-c5f1-440a-9bca-e1f27c19d2f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:08.198845 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6277b9-c5f1-440a-9bca-e1f27c19d2f4 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:08.198985 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1008/2024] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:08 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:08.199397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6277b9-c5f1-440a-9bca-e1f27c19d2f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:08.199397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6277b9-c5f1-440a-9bca-e1f27c19d2f4 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:08.200381 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6277b9-c5f1-440a-9bca-e1f27c19d2f4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:08.202358 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6277b9-c5f1-440a-9bca-e1f27c19d2f4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:08.203176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df66fb7-11e3-470e-8e46-578d9d0ef705 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:08.214611 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df66fb7-11e3-470e-8e46-578d9d0ef705 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:08.220113 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578bcd9f-1283-47ef-82c2-a1c78e0a1617 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:08.220715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578bcd9f-1283-47ef-82c2-a1c78e0a1617 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:08.225596 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578bcd9f-1283-47ef-82c2-a1c78e0a1617 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:08.225596 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578bcd9f-1283-47ef-82c2-a1c78e0a1617 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:08.225596 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578bcd9f-1283-47ef-82c2-a1c78e0a1617 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:08.225596 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578bcd9f-1283-47ef-82c2-a1c78e0a1617 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:08.238779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1009/2025] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:08 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:08.238779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578bcd9f-1283-47ef-82c2-a1c78e0a1617 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:08.251179 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578bcd9f-1283-47ef-82c2-a1c78e0a1617 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:08.257140 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0dbb7f-fac1-44c6-ac56-a7af384bb976 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:08.257983 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0dbb7f-fac1-44c6-ac56-a7af384bb976 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:08.258286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0dbb7f-fac1-44c6-ac56-a7af384bb976 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:08.258717 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0dbb7f-fac1-44c6-ac56-a7af384bb976 admin admin] PATH_INFO: `/v3/projects/bb7164960bf24f24b7aed86e9e40130e/users/f06ff49709e04623b42db4b0c7612163/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:08.279702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0dbb7f-fac1-44c6-ac56-a7af384bb976 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb7164960bf24f24b7aed86e9e40130e, user_id=f06ff49709e04623b42db4b0c7612163, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:08.282489 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0dbb7f-fac1-44c6-ac56-a7af384bb976 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:08.370769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1017/2026] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:08 2022] POST /identity/v3/users => generated 299 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:08.371729 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6277b9-c5f1-440a-9bca-e1f27c19d2f4 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:08.398289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1010/2027] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:08 2022] PUT /identity/v3/projects/bb7164960bf24f24b7aed86e9e40130e/users/f06ff49709e04623b42db4b0c7612163/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:08.408316 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6277b9-c5f1-440a-9bca-e1f27c19d2f4 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:08.410413 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0dbb7f-fac1-44c6-ac56-a7af384bb976 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:08.470683 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:08.486123 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a5a994-21b6-4bb3-b6eb-89c8c0ee959f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:08.486830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a5a994-21b6-4bb3-b6eb-89c8c0ee959f admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:08.487113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a5a994-21b6-4bb3-b6eb-89c8c0ee959f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:08.487533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a5a994-21b6-4bb3-b6eb-89c8c0ee959f admin admin] PATH_INFO: `/v3/projects/ac90a94a900543c7b7017f2a053a9fda/users/c741b5caba1c42a5a7130f3b69be90f1/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:08.493914 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0dbb7f-fac1-44c6-ac56-a7af384bb976 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:08.499111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c7cb99-f81c-4eca-8540-7e82c263ac47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:08.499958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c7cb99-f81c-4eca-8540-7e82c263ac47 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:08.500087 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c7cb99-f81c-4eca-8540-7e82c263ac47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:08.500479 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c7cb99-f81c-4eca-8540-7e82c263ac47 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:08.501124 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c7cb99-f81c-4eca-8540-7e82c263ac47 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:08.502651 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a5a994-21b6-4bb3-b6eb-89c8c0ee959f admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac90a94a900543c7b7017f2a053a9fda, user_id=c741b5caba1c42a5a7130f3b69be90f1, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:08.502796 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c7cb99-f81c-4eca-8540-7e82c263ac47 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:08.505304 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a5a994-21b6-4bb3-b6eb-89c8c0ee959f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:08.512645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1011/2028] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:08 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:08.514348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e6eca3-ed47-4038-92e6-94bc05b0c163 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:08.514651 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e6eca3-ed47-4038-92e6-94bc05b0c163 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:08.514926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e6eca3-ed47-4038-92e6-94bc05b0c163 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:08.541832 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-48e6eca3-ed47-4038-92e6-94bc05b0c163 None None] MFA Rules not processed for user `f06ff49709e04623b42db4b0c7612163`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:08.543058 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-48e6eca3-ed47-4038-92e6-94bc05b0c163 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:08.604117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1012/2029] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:08 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:08.604883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e6eca3-ed47-4038-92e6-94bc05b0c163 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:08.614547 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e6eca3-ed47-4038-92e6-94bc05b0c163 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:08.619371 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427281e3-b740-4f86-b44f-c75fd4587ac4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:08.620263 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427281e3-b740-4f86-b44f-c75fd4587ac4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:08.620540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427281e3-b740-4f86-b44f-c75fd4587ac4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:08.620880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427281e3-b740-4f86-b44f-c75fd4587ac4 admin admin] PATH_INFO: `/v3/projects/d951df5495244097a4026510af3b616f/users/1ec537f4d1a64a4e9330d3cec97e513e/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:08.632731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1018/2030] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:08 2022] PUT /identity/v3/projects/ac90a94a900543c7b7017f2a053a9fda/users/c741b5caba1c42a5a7130f3b69be90f1/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:08.632731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a5a994-21b6-4bb3-b6eb-89c8c0ee959f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:08.640091 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427281e3-b740-4f86-b44f-c75fd4587ac4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d951df5495244097a4026510af3b616f, user_id=1ec537f4d1a64a4e9330d3cec97e513e, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:08.642936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427281e3-b740-4f86-b44f-c75fd4587ac4 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:08.694816 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a5a994-21b6-4bb3-b6eb-89c8c0ee959f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:08.699514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730c9892-ffde-4601-b621-7582f61469c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:08.700125 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730c9892-ffde-4601-b621-7582f61469c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:08.700907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730c9892-ffde-4601-b621-7582f61469c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:08.701188 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730c9892-ffde-4601-b621-7582f61469c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:08.701820 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730c9892-ffde-4601-b621-7582f61469c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:08.703788 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730c9892-ffde-4601-b621-7582f61469c6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:08.737927 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1013/2031] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:08 2022] PUT /identity/v3/projects/d951df5495244097a4026510af3b616f/users/1ec537f4d1a64a4e9330d3cec97e513e/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:08.738812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427281e3-b740-4f86-b44f-c75fd4587ac4 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:08.789977 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1019/2032] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:08 2022] POST /identity/v3/projects => generated 387 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:08.791252 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730c9892-ffde-4601-b621-7582f61469c6 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:08.806675 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427281e3-b740-4f86-b44f-c75fd4587ac4 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:08.818351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:08.857081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730c9892-ffde-4601-b621-7582f61469c6 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:08.862181 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd0367e-c8d4-4258-ab9d-c885a8e6bd6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:08.862826 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd0367e-c8d4-4258-ab9d-c885a8e6bd6a admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:08.863110 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd0367e-c8d4-4258-ab9d-c885a8e6bd6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:08.863379 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd0367e-c8d4-4258-ab9d-c885a8e6bd6a admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:08.864018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd0367e-c8d4-4258-ab9d-c885a8e6bd6a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:08.864350 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272b9aa9-6175-40c0-9a02-ba2764144edd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:08.864964 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272b9aa9-6175-40c0-9a02-ba2764144edd admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:08.865303 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272b9aa9-6175-40c0-9a02-ba2764144edd admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:08.865717 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272b9aa9-6175-40c0-9a02-ba2764144edd admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:08.865835 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd0367e-c8d4-4258-ab9d-c885a8e6bd6a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:08.866217 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-272b9aa9-6175-40c0-9a02-ba2764144edd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:08.867865 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-272b9aa9-6175-40c0-9a02-ba2764144edd admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:08.879856 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1020/2033] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:08 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:08.879856 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd0367e-c8d4-4258-ab9d-c885a8e6bd6a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:08.880138 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1014/2034] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:08 2022] GET /identity/v3/roles => generated 1420 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:08.880906 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272b9aa9-6175-40c0-9a02-ba2764144edd admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:08.890098 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd0367e-c8d4-4258-ab9d-c885a8e6bd6a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:08.891354 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272b9aa9-6175-40c0-9a02-ba2764144edd admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:08.894986 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364cc8b3-179c-472f-a594-849b6d8e9a8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:08.895580 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364cc8b3-179c-472f-a594-849b6d8e9a8a admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:08.895861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364cc8b3-179c-472f-a594-849b6d8e9a8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:08.896166 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364cc8b3-179c-472f-a594-849b6d8e9a8a admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:08.896792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364cc8b3-179c-472f-a594-849b6d8e9a8a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:08.897449 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:08.898288 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57592f88-ec8e-43c0-9991-a0f943535751 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:08.898745 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364cc8b3-179c-472f-a594-849b6d8e9a8a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:08.899112 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57592f88-ec8e-43c0-9991-a0f943535751 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:08.899433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57592f88-ec8e-43c0-9991-a0f943535751 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:08.899737 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57592f88-ec8e-43c0-9991-a0f943535751 admin admin] PATH_INFO: `/v3/projects/d951df5495244097a4026510af3b616f/users/1ec537f4d1a64a4e9330d3cec97e513e/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:08.901787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57592f88-ec8e-43c0-9991-a0f943535751 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d951df5495244097a4026510af3b616f, user_id=1ec537f4d1a64a4e9330d3cec97e513e, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:08.904583 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57592f88-ec8e-43c0-9991-a0f943535751 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:08.915053 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1015/2035] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:08 2022] PUT /identity/v3/projects/d951df5495244097a4026510af3b616f/users/1ec537f4d1a64a4e9330d3cec97e513e/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:08.916467 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57592f88-ec8e-43c0-9991-a0f943535751 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:08.976971 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57592f88-ec8e-43c0-9991-a0f943535751 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:08.981701 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec1bfcc-8c32-48ff-a352-8c534f171a40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:08.982348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec1bfcc-8c32-48ff-a352-8c534f171a40 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:08.982653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec1bfcc-8c32-48ff-a352-8c534f171a40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:08.982925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec1bfcc-8c32-48ff-a352-8c534f171a40 admin admin] PATH_INFO: `/v3/projects/ac90a94a900543c7b7017f2a053a9fda/users/c741b5caba1c42a5a7130f3b69be90f1/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:08.984828 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec1bfcc-8c32-48ff-a352-8c534f171a40 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac90a94a900543c7b7017f2a053a9fda, user_id=c741b5caba1c42a5a7130f3b69be90f1, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:08.987506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec1bfcc-8c32-48ff-a352-8c534f171a40 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:08.997966 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1016/2036] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:08 2022] PUT /identity/v3/projects/ac90a94a900543c7b7017f2a053a9fda/users/c741b5caba1c42a5a7130f3b69be90f1/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:08.999104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4abe9b-38f9-4198-b205-3f31587c95b8 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:08.999387 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4abe9b-38f9-4198-b205-3f31587c95b8 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:08.999654 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4abe9b-38f9-4198-b205-3f31587c95b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:09.012266 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1021/2037] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:08 2022] POST /identity/v3/users => generated 300 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:09.014338 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27092513-4c4e-41fd-ae78-29598226eaf7 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:09.014638 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27092513-4c4e-41fd-ae78-29598226eaf7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:09.014910 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27092513-4c4e-41fd-ae78-29598226eaf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:09.033511 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-ff4abe9b-38f9-4198-b205-3f31587c95b8 None None] MFA Rules not processed for user `1ec537f4d1a64a4e9330d3cec97e513e`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:09.034783 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ff4abe9b-38f9-4198-b205-3f31587c95b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:09.044608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-27092513-4c4e-41fd-ae78-29598226eaf7 None None] MFA Rules not processed for user `c741b5caba1c42a5a7130f3b69be90f1`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:09.046519 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-27092513-4c4e-41fd-ae78-29598226eaf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:09.117913 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1017/2038] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:08 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:09.118723 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4abe9b-38f9-4198-b205-3f31587c95b8 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:09.120418 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1022/2039] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:09 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:09.135209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27092513-4c4e-41fd-ae78-29598226eaf7 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:09.137816 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-27092513-4c4e-41fd-ae78-29598226eaf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:09.193983 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4abe9b-38f9-4198-b205-3f31587c95b8 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:09.199057 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec44fad3-5dac-4578-862f-224b6b2ae3fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:09.199781 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec44fad3-5dac-4578-862f-224b6b2ae3fb admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:09.200298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec44fad3-5dac-4578-862f-224b6b2ae3fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:09.200402 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27092513-4c4e-41fd-ae78-29598226eaf7 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:09.200705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec44fad3-5dac-4578-862f-224b6b2ae3fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:09.201480 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec44fad3-5dac-4578-862f-224b6b2ae3fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:09.203127 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec44fad3-5dac-4578-862f-224b6b2ae3fb admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:09.204195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b642874-895e-4b23-8f93-ef20434e8134 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:09.204839 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b642874-895e-4b23-8f93-ef20434e8134 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:09.205120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b642874-895e-4b23-8f93-ef20434e8134 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:09.205395 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b642874-895e-4b23-8f93-ef20434e8134 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:09.216282 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1018/2040] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:09 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:09.216282 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec44fad3-5dac-4578-862f-224b6b2ae3fb admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:09.226955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec44fad3-5dac-4578-862f-224b6b2ae3fb admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:09.231281 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e195381-be02-4be0-81a9-c013ed89f881 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:09.232502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e195381-be02-4be0-81a9-c013ed89f881 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:09.233151 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e195381-be02-4be0-81a9-c013ed89f881 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:09.233733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e195381-be02-4be0-81a9-c013ed89f881 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:09.235719 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:09.245319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e195381-be02-4be0-81a9-c013ed89f881 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:09.246786 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e195381-be02-4be0-81a9-c013ed89f881 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:09.251138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:09.256961 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1019/2041] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:09 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:09.257730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e195381-be02-4be0-81a9-c013ed89f881 service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:09.269133 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b642874-895e-4b23-8f93-ef20434e8134 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:09.270583 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b642874-895e-4b23-8f93-ef20434e8134 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:09.271049 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e195381-be02-4be0-81a9-c013ed89f881 service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:09.277811 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698b8a45-561b-4888-9e0f-35ace76f10e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:09.278613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698b8a45-561b-4888-9e0f-35ace76f10e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:09.279014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698b8a45-561b-4888-9e0f-35ace76f10e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:09.279359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698b8a45-561b-4888-9e0f-35ace76f10e6 admin admin] PATH_INFO: `/v3/projects/578897d982ff44169a17f9d384c801d1/users/3050bf0a0b014f6db88a28a5963b9de8/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:09.282220 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1023/2042] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:09 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:47:09.296767 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698b8a45-561b-4888-9e0f-35ace76f10e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=578897d982ff44169a17f9d384c801d1, user_id=3050bf0a0b014f6db88a28a5963b9de8, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:09.299653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698b8a45-561b-4888-9e0f-35ace76f10e6 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:09.374088 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1020/2043] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:09 2022] PUT /identity/v3/projects/578897d982ff44169a17f9d384c801d1/users/3050bf0a0b014f6db88a28a5963b9de8/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:09.378220 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b642874-895e-4b23-8f93-ef20434e8134 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:09.390517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b642874-895e-4b23-8f93-ef20434e8134 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:09.462416 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf979abc-7c0f-48e5-bf21-33c902a85f7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:09.463221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf979abc-7c0f-48e5-bf21-33c902a85f7d admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:09.463610 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf979abc-7c0f-48e5-bf21-33c902a85f7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:09.463951 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf979abc-7c0f-48e5-bf21-33c902a85f7d admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:09.464676 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf979abc-7c0f-48e5-bf21-33c902a85f7d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:09.466392 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf979abc-7c0f-48e5-bf21-33c902a85f7d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:09.476987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1024/2044] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:09 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:47:09.481247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698b8a45-561b-4888-9e0f-35ace76f10e6 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:09.492866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698b8a45-561b-4888-9e0f-35ace76f10e6 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:09.499715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fff315-d622-49ab-8dca-7e818644bbaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:09.500407 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fff315-d622-49ab-8dca-7e818644bbaf admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:09.500689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fff315-d622-49ab-8dca-7e818644bbaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:09.500971 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fff315-d622-49ab-8dca-7e818644bbaf admin admin] PATH_INFO: `/v3/projects/578897d982ff44169a17f9d384c801d1/users/3050bf0a0b014f6db88a28a5963b9de8/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:09.503062 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fff315-d622-49ab-8dca-7e818644bbaf admin admin] RBAC: Authorizing `identity:create_grant(project_id=578897d982ff44169a17f9d384c801d1, user_id=3050bf0a0b014f6db88a28a5963b9de8, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:09.505861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fff315-d622-49ab-8dca-7e818644bbaf admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:09.518317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1021/2045] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:09 2022] PUT /identity/v3/projects/578897d982ff44169a17f9d384c801d1/users/3050bf0a0b014f6db88a28a5963b9de8/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:09.529180 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6231f1d6-5c29-42be-aaa4-22eb09a508da None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:09.529678 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6231f1d6-5c29-42be-aaa4-22eb09a508da None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:09.529773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6231f1d6-5c29-42be-aaa4-22eb09a508da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:09.563418 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-6231f1d6-5c29-42be-aaa4-22eb09a508da None None] MFA Rules not processed for user `3050bf0a0b014f6db88a28a5963b9de8`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:09.564965 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6231f1d6-5c29-42be-aaa4-22eb09a508da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:09.657493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1025/2046] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:09 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:09.688532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fff315-d622-49ab-8dca-7e818644bbaf admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:09.746871 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fff315-d622-49ab-8dca-7e818644bbaf admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:09.751368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1a4313-fe40-4f1d-ae69-af93f8f81c6d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:09.751986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1a4313-fe40-4f1d-ae69-af93f8f81c6d service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:09.752266 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1a4313-fe40-4f1d-ae69-af93f8f81c6d service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:09.752532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1a4313-fe40-4f1d-ae69-af93f8f81c6d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:09.814401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1a4313-fe40-4f1d-ae69-af93f8f81c6d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:09.815988 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1a4313-fe40-4f1d-ae69-af93f8f81c6d service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:09.826263 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1022/2047] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:09 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:10.210415 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6231f1d6-5c29-42be-aaa4-22eb09a508da None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:10.220410 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6231f1d6-5c29-42be-aaa4-22eb09a508da None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:10.224073 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ab693c-a478-4c92-940c-08a5798ddf8a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:10.224782 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ab693c-a478-4c92-940c-08a5798ddf8a service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:10.225138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ab693c-a478-4c92-940c-08a5798ddf8a service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:10.225467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ab693c-a478-4c92-940c-08a5798ddf8a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:10.235295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ab693c-a478-4c92-940c-08a5798ddf8a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:10.236846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ab693c-a478-4c92-940c-08a5798ddf8a service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:10.249128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1026/2048] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:10 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:11.756983 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1a4313-fe40-4f1d-ae69-af93f8f81c6d service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:11.766246 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1a4313-fe40-4f1d-ae69-af93f8f81c6d service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:11.770271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfd4a20-c8b5-4e84-bf6d-5d05e9933c7e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:11.771149 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfd4a20-c8b5-4e84-bf6d-5d05e9933c7e service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:11.771592 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfd4a20-c8b5-4e84-bf6d-5d05e9933c7e service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:11.771892 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfd4a20-c8b5-4e84-bf6d-5d05e9933c7e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:11.813467 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:11.842732 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfd4a20-c8b5-4e84-bf6d-5d05e9933c7e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:11.844194 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfd4a20-c8b5-4e84-bf6d-5d05e9933c7e service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:11.854484 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1023/2049] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:11 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:11.866623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ab693c-a478-4c92-940c-08a5798ddf8a service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:11.876002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ab693c-a478-4c92-940c-08a5798ddf8a service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:11.880044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3191a979-7039-4ff7-bd45-328e7f26a5c9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:11.880717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3191a979-7039-4ff7-bd45-328e7f26a5c9 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:11.881070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3191a979-7039-4ff7-bd45-328e7f26a5c9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:11.881422 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3191a979-7039-4ff7-bd45-328e7f26a5c9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:11.934557 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:11.950314 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3191a979-7039-4ff7-bd45-328e7f26a5c9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:11.952169 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3191a979-7039-4ff7-bd45-328e7f26a5c9 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:11.961987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1027/2050] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:11 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:13.473876 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfd4a20-c8b5-4e84-bf6d-5d05e9933c7e service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:13.483205 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfd4a20-c8b5-4e84-bf6d-5d05e9933c7e service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:13.486875 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8b294f-9496-4902-ab61-57bc52a0c3ed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:13.487486 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8b294f-9496-4902-ab61-57bc52a0c3ed service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:13.487767 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8b294f-9496-4902-ab61-57bc52a0c3ed service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:13.488038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8b294f-9496-4902-ab61-57bc52a0c3ed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:13.578575 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8b294f-9496-4902-ab61-57bc52a0c3ed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:13.580052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8b294f-9496-4902-ab61-57bc52a0c3ed service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:13.589920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1024/2051] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:13 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:14.621778 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3191a979-7039-4ff7-bd45-328e7f26a5c9 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:14.703730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3191a979-7039-4ff7-bd45-328e7f26a5c9 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:14.710011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734fbc9b-2131-4e13-a80d-ea414904e106 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:14.710944 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734fbc9b-2131-4e13-a80d-ea414904e106 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:14.711426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734fbc9b-2131-4e13-a80d-ea414904e106 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:14.711812 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734fbc9b-2131-4e13-a80d-ea414904e106 admin admin] PATH_INFO: `/v3/users/3372a4db7fae497cb31be248aad12329` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:14.713289 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734fbc9b-2131-4e13-a80d-ea414904e106 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3372a4db7fae497cb31be248aad12329)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:14.716403 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734fbc9b-2131-4e13-a80d-ea414904e106 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:14.866523 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8b294f-9496-4902-ab61-57bc52a0c3ed service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:14.877495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8b294f-9496-4902-ab61-57bc52a0c3ed service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:14.882068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fa28b0-d266-43f4-a635-21dd2eae32e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:14.882727 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fa28b0-d266-43f4-a635-21dd2eae32e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:14.883023 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fa28b0-d266-43f4-a635-21dd2eae32e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:14.883297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fa28b0-d266-43f4-a635-21dd2eae32e1 admin admin] PATH_INFO: `/v3/users/4c7ee08a9a4848698ac7e1c21f08d5e0` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:14.884404 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fa28b0-d266-43f4-a635-21dd2eae32e1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4c7ee08a9a4848698ac7e1c21f08d5e0)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:14.886342 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fa28b0-d266-43f4-a635-21dd2eae32e1 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:14.907026 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-734fbc9b-2131-4e13-a80d-ea414904e106 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3372a4db7fae497cb31be248aad12329'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:14.907726 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-734fbc9b-2131-4e13-a80d-ea414904e106 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3372a4db7fae497cb31be248aad12329'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:14.908341 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-734fbc9b-2131-4e13-a80d-ea414904e106 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3372a4db7fae497cb31be248aad12329'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:14.920356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-734fbc9b-2131-4e13-a80d-ea414904e106 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3372a4db7fae497cb31be248aad12329'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:14.976422 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-734fbc9b-2131-4e13-a80d-ea414904e106 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3372a4db7fae497cb31be248aad12329'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:14.997382 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1028/2052] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:14 2022] DELETE /identity/v3/users/3372a4db7fae497cb31be248aad12329 => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:15.002928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c4a5d5-cec9-4aad-a32c-caf03d0688f0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:15.003324 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c4a5d5-cec9-4aad-a32c-caf03d0688f0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:15.003817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c4a5d5-cec9-4aad-a32c-caf03d0688f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:15.024435 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e8c4a5d5-cec9-4aad-a32c-caf03d0688f0 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:15.025658 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e8c4a5d5-cec9-4aad-a32c-caf03d0688f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:15.089449 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e1fa28b0-d266-43f4-a635-21dd2eae32e1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4c7ee08a9a4848698ac7e1c21f08d5e0'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:15.090926 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1029/2053] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:47:15.101250 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c4a5d5-cec9-4aad-a32c-caf03d0688f0 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:15.103018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e8c4a5d5-cec9-4aad-a32c-caf03d0688f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:15.106135 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e1fa28b0-d266-43f4-a635-21dd2eae32e1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4c7ee08a9a4848698ac7e1c21f08d5e0'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:15.106961 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e1fa28b0-d266-43f4-a635-21dd2eae32e1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4c7ee08a9a4848698ac7e1c21f08d5e0'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:15.116372 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:15.173501 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c4a5d5-cec9-4aad-a32c-caf03d0688f0 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:15.178384 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c97c064-1afc-454e-ba67-5c3e77fdb81e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:15.179633 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c97c064-1afc-454e-ba67-5c3e77fdb81e service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:15.179746 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c97c064-1afc-454e-ba67-5c3e77fdb81e service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:15.180099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c97c064-1afc-454e-ba67-5c3e77fdb81e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:15.180536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e1fa28b0-d266-43f4-a635-21dd2eae32e1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4c7ee08a9a4848698ac7e1c21f08d5e0'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:15.181985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9c97c064-1afc-454e-ba67-5c3e77fdb81e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:15.191374 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e1fa28b0-d266-43f4-a635-21dd2eae32e1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4c7ee08a9a4848698ac7e1c21f08d5e0'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:15.206042 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c97c064-1afc-454e-ba67-5c3e77fdb81e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:15.212831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1025/2054] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:14 2022] DELETE /identity/v3/users/4c7ee08a9a4848698ac7e1c21f08d5e0 => generated 0 bytes in 341 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:47:15.212973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c97c064-1afc-454e-ba67-5c3e77fdb81e service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:15.213070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b80f85-2745-42d6-bb51-6ee985d88481 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:15.213070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b80f85-2745-42d6-bb51-6ee985d88481 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:15.213070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b80f85-2745-42d6-bb51-6ee985d88481 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:15.218422 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1030/2055] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:15.235388 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-b0b80f85-2745-42d6-bb51-6ee985d88481 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:15.236829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b0b80f85-2745-42d6-bb51-6ee985d88481 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:15.271884 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:15.301366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1026/2056] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:47:15.312721 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c97c064-1afc-454e-ba67-5c3e77fdb81e service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:15.314267 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9c97c064-1afc-454e-ba67-5c3e77fdb81e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:15.325192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c97c064-1afc-454e-ba67-5c3e77fdb81e service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:15.385471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753501c2-1f5e-4b3f-9f67-115ef242f136 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:15.385471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753501c2-1f5e-4b3f-9f67-115ef242f136 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:15.385471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753501c2-1f5e-4b3f-9f67-115ef242f136 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:15.385471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753501c2-1f5e-4b3f-9f67-115ef242f136 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:15.393221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753501c2-1f5e-4b3f-9f67-115ef242f136 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:15.394693 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753501c2-1f5e-4b3f-9f67-115ef242f136 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:15.404325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1031/2057] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:15.774308 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b80f85-2745-42d6-bb51-6ee985d88481 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:15.782291 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b0b80f85-2745-42d6-bb51-6ee985d88481 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:15.818919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b80f85-2745-42d6-bb51-6ee985d88481 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:15.851333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6978f006-2b9c-417d-97fa-0f281747db2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:15.851333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6978f006-2b9c-417d-97fa-0f281747db2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:15.851333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6978f006-2b9c-417d-97fa-0f281747db2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:15.851333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6978f006-2b9c-417d-97fa-0f281747db2c admin admin] PATH_INFO: `/v3/projects/aa4000b6511c476eb95622a6c86f4cf7` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:15.851673 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6978f006-2b9c-417d-97fa-0f281747db2c admin admin] RBAC: Authorizing `identity:delete_project(project_id=aa4000b6511c476eb95622a6c86f4cf7)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:15.853973 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6978f006-2b9c-417d-97fa-0f281747db2c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:15.965335 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6978f006-2b9c-417d-97fa-0f281747db2c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aa4000b6511c476eb95622a6c86f4cf7', 'request_id': 'req-6978f006-2b9c-417d-97fa-0f281747db2c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:15.980478 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6978f006-2b9c-417d-97fa-0f281747db2c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aa4000b6511c476eb95622a6c86f4cf7', 'request_id': 'req-6978f006-2b9c-417d-97fa-0f281747db2c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:15.984130 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6978f006-2b9c-417d-97fa-0f281747db2c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aa4000b6511c476eb95622a6c86f4cf7', 'request_id': 'req-6978f006-2b9c-417d-97fa-0f281747db2c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:16.036419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6978f006-2b9c-417d-97fa-0f281747db2c admin admin] The token cache is being invalidate because project aa4000b6511c476eb95622a6c86f4cf7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:16.036840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6978f006-2b9c-417d-97fa-0f281747db2c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:16.037585 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6978f006-2b9c-417d-97fa-0f281747db2c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:16.039378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1027/2058] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:15 2022] DELETE /identity/v3/projects/aa4000b6511c476eb95622a6c86f4cf7 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:16.043417 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753501c2-1f5e-4b3f-9f67-115ef242f136 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:16.054867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753501c2-1f5e-4b3f-9f67-115ef242f136 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:16.127086 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1befa9b9-8bcc-4b16-899d-066a558dc2c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:16.127890 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1befa9b9-8bcc-4b16-899d-066a558dc2c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:16.128274 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1befa9b9-8bcc-4b16-899d-066a558dc2c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:16.128616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1befa9b9-8bcc-4b16-899d-066a558dc2c1 admin admin] PATH_INFO: `/v3/users/dfbe496c221648908e1cc158ef6f10c5` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:16.129864 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1befa9b9-8bcc-4b16-899d-066a558dc2c1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dfbe496c221648908e1cc158ef6f10c5)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:16.132448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1befa9b9-8bcc-4b16-899d-066a558dc2c1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:16.321798 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6978f006-2b9c-417d-97fa-0f281747db2c admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:16.323295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-6978f006-2b9c-417d-97fa-0f281747db2c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:16.339520 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6978f006-2b9c-417d-97fa-0f281747db2c admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:16.343106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1befa9b9-8bcc-4b16-899d-066a558dc2c1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dfbe496c221648908e1cc158ef6f10c5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:16.344308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1befa9b9-8bcc-4b16-899d-066a558dc2c1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dfbe496c221648908e1cc158ef6f10c5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:16.344994 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1befa9b9-8bcc-4b16-899d-066a558dc2c1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dfbe496c221648908e1cc158ef6f10c5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:16.346049 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3bd585-5e6a-4d38-b232-e735d9f3f673 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:16.346731 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3bd585-5e6a-4d38-b232-e735d9f3f673 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:16.347016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3bd585-5e6a-4d38-b232-e735d9f3f673 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:16.347398 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3bd585-5e6a-4d38-b232-e735d9f3f673 admin admin] PATH_INFO: `/v3/projects/e8f3f11ba4544c259c7fb8c6012443cb` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:16.348624 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3bd585-5e6a-4d38-b232-e735d9f3f673 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e8f3f11ba4544c259c7fb8c6012443cb)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:16.350879 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3bd585-5e6a-4d38-b232-e735d9f3f673 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:16.352070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:16.355736 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1befa9b9-8bcc-4b16-899d-066a558dc2c1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dfbe496c221648908e1cc158ef6f10c5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:16.411839 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1befa9b9-8bcc-4b16-899d-066a558dc2c1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dfbe496c221648908e1cc158ef6f10c5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:16.429904 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1032/2059] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:16 2022] DELETE /identity/v3/users/dfbe496c221648908e1cc158ef6f10c5 => generated 0 bytes in 387 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:16.453503 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6f3bd585-5e6a-4d38-b232-e735d9f3f673 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e8f3f11ba4544c259c7fb8c6012443cb', 'request_id': 'req-6f3bd585-5e6a-4d38-b232-e735d9f3f673'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:16.459406 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6f3bd585-5e6a-4d38-b232-e735d9f3f673 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e8f3f11ba4544c259c7fb8c6012443cb', 'request_id': 'req-6f3bd585-5e6a-4d38-b232-e735d9f3f673'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:16.466043 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6f3bd585-5e6a-4d38-b232-e735d9f3f673 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e8f3f11ba4544c259c7fb8c6012443cb', 'request_id': 'req-6f3bd585-5e6a-4d38-b232-e735d9f3f673'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:16.561017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6f3bd585-5e6a-4d38-b232-e735d9f3f673 admin admin] The token cache is being invalidate because project e8f3f11ba4544c259c7fb8c6012443cb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:16.561315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6f3bd585-5e6a-4d38-b232-e735d9f3f673 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:16.561946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6f3bd585-5e6a-4d38-b232-e735d9f3f673 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:16.563541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1028/2060] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:16 2022] DELETE /identity/v3/projects/e8f3f11ba4544c259c7fb8c6012443cb => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:47:16.567500 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1befa9b9-8bcc-4b16-899d-066a558dc2c1 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:16.568919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1befa9b9-8bcc-4b16-899d-066a558dc2c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:16.670402 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1befa9b9-8bcc-4b16-899d-066a558dc2c1 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:16.742928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a63341b-ad5c-4988-9679-9836b0514dcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:16.743441 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a63341b-ad5c-4988-9679-9836b0514dcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:16.743879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a63341b-ad5c-4988-9679-9836b0514dcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:16.744187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a63341b-ad5c-4988-9679-9836b0514dcb admin admin] PATH_INFO: `/v3/users/c741b5caba1c42a5a7130f3b69be90f1` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:16.747908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a63341b-ad5c-4988-9679-9836b0514dcb admin admin] RBAC: Authorizing `identity:delete_user(user_id=c741b5caba1c42a5a7130f3b69be90f1)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:16.749993 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a63341b-ad5c-4988-9679-9836b0514dcb admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:16.907371 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3bd585-5e6a-4d38-b232-e735d9f3f673 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:16.908919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-6f3bd585-5e6a-4d38-b232-e735d9f3f673 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:16.933341 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3bd585-5e6a-4d38-b232-e735d9f3f673 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:16.938475 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df55e900-7781-4f2c-9163-17e2e5366887 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:16.939204 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df55e900-7781-4f2c-9163-17e2e5366887 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:16.941239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df55e900-7781-4f2c-9163-17e2e5366887 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:16.941566 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df55e900-7781-4f2c-9163-17e2e5366887 admin admin] PATH_INFO: `/v3/projects/014647dd60d54c4791f979e379e6be42` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:16.942812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df55e900-7781-4f2c-9163-17e2e5366887 admin admin] RBAC: Authorizing `identity:delete_project(project_id=014647dd60d54c4791f979e379e6be42)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:16.944979 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df55e900-7781-4f2c-9163-17e2e5366887 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:16.992123 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4a63341b-ad5c-4988-9679-9836b0514dcb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c741b5caba1c42a5a7130f3b69be90f1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:17.004204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4a63341b-ad5c-4988-9679-9836b0514dcb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c741b5caba1c42a5a7130f3b69be90f1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:17.007046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4a63341b-ad5c-4988-9679-9836b0514dcb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c741b5caba1c42a5a7130f3b69be90f1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:17.024420 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4a63341b-ad5c-4988-9679-9836b0514dcb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c741b5caba1c42a5a7130f3b69be90f1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:17.095847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4a63341b-ad5c-4988-9679-9836b0514dcb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c741b5caba1c42a5a7130f3b69be90f1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:17.106026 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:17.112885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=85,keystone:INSERT=2,keystone:DELETE=13 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:17.115630 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-df55e900-7781-4f2c-9163-17e2e5366887 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '014647dd60d54c4791f979e379e6be42', 'request_id': 'req-df55e900-7781-4f2c-9163-17e2e5366887'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:17.118070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1033/2061] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:16 2022] DELETE /identity/v3/users/c741b5caba1c42a5a7130f3b69be90f1 => generated 0 bytes in 551 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:17.130827 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-df55e900-7781-4f2c-9163-17e2e5366887 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '014647dd60d54c4791f979e379e6be42', 'request_id': 'req-df55e900-7781-4f2c-9163-17e2e5366887'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:17.142209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-df55e900-7781-4f2c-9163-17e2e5366887 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '014647dd60d54c4791f979e379e6be42', 'request_id': 'req-df55e900-7781-4f2c-9163-17e2e5366887'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:17.234974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-df55e900-7781-4f2c-9163-17e2e5366887 admin admin] The token cache is being invalidate because project 014647dd60d54c4791f979e379e6be42 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:17.234974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-df55e900-7781-4f2c-9163-17e2e5366887 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:17.234974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-df55e900-7781-4f2c-9163-17e2e5366887 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:17.234974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1029/2062] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:16 2022] DELETE /identity/v3/projects/014647dd60d54c4791f979e379e6be42 => generated 0 bytes in 325 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:17.241933 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a4b0ff-4993-47ea-a6c7-8f872329ba77 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:17.242228 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a4b0ff-4993-47ea-a6c7-8f872329ba77 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:17.242508 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a4b0ff-4993-47ea-a6c7-8f872329ba77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:17.295021 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-86a4b0ff-4993-47ea-a6c7-8f872329ba77 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:17.296432 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-86a4b0ff-4993-47ea-a6c7-8f872329ba77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:17.367682 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1034/2063] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:17.376499 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f591437-ab5f-4a65-9278-94cdda85a8e8 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:17.376797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f591437-ab5f-4a65-9278-94cdda85a8e8 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:17.377060 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f591437-ab5f-4a65-9278-94cdda85a8e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:17.410947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-8f591437-ab5f-4a65-9278-94cdda85a8e8 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:17.412338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-8f591437-ab5f-4a65-9278-94cdda85a8e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:17.422737 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1030/2064] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:17.436945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a4b0ff-4993-47ea-a6c7-8f872329ba77 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:17.438690 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-86a4b0ff-4993-47ea-a6c7-8f872329ba77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:17.441624 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f591437-ab5f-4a65-9278-94cdda85a8e8 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:17.454262 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f591437-ab5f-4a65-9278-94cdda85a8e8 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:17.462953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b160ca1-2ffe-4bea-82da-39c876b62067 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:17.463501 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b160ca1-2ffe-4bea-82da-39c876b62067 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:17.464531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b160ca1-2ffe-4bea-82da-39c876b62067 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:17.464717 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b160ca1-2ffe-4bea-82da-39c876b62067 admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:17.465357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b160ca1-2ffe-4bea-82da-39c876b62067 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:47:17.465775 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b160ca1-2ffe-4bea-82da-39c876b62067 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:17.467654 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b160ca1-2ffe-4bea-82da-39c876b62067 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:17.485945 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1031/2065] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:47:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:47:17.488043 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b160ca1-2ffe-4bea-82da-39c876b62067 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:17.506384 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b160ca1-2ffe-4bea-82da-39c876b62067 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:17.514451 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa542d8b-894b-4802-9b77-10c9600da19c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:17.515668 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa542d8b-894b-4802-9b77-10c9600da19c admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:17.515956 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa542d8b-894b-4802-9b77-10c9600da19c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:17.516613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa542d8b-894b-4802-9b77-10c9600da19c admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:17.516922 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa542d8b-894b-4802-9b77-10c9600da19c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:17.519656 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa542d8b-894b-4802-9b77-10c9600da19c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:17.529634 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a4b0ff-4993-47ea-a6c7-8f872329ba77 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:17.534222 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61e58a4-9c3c-43ea-8aeb-519783bcc0fd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:17.535341 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61e58a4-9c3c-43ea-8aeb-519783bcc0fd service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:17.543043 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61e58a4-9c3c-43ea-8aeb-519783bcc0fd service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:17.543509 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61e58a4-9c3c-43ea-8aeb-519783bcc0fd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:17.546197 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d61e58a4-9c3c-43ea-8aeb-519783bcc0fd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:17.610744 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61e58a4-9c3c-43ea-8aeb-519783bcc0fd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:17.612207 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61e58a4-9c3c-43ea-8aeb-519783bcc0fd service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:17.621600 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1035/2066] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:17 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:17.663586 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61e58a4-9c3c-43ea-8aeb-519783bcc0fd service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:17.663846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1032/2067] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:17 2022] POST /identity/v3/projects => generated 385 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:17.667942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa542d8b-894b-4802-9b77-10c9600da19c admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:17.669405 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d61e58a4-9c3c-43ea-8aeb-519783bcc0fd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:17.691565 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61e58a4-9c3c-43ea-8aeb-519783bcc0fd service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:17.748532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa542d8b-894b-4802-9b77-10c9600da19c admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:17.751974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455784b8-082c-49d2-8835-c9756d1912ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:17.752629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455784b8-082c-49d2-8835-c9756d1912ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:17.752944 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455784b8-082c-49d2-8835-c9756d1912ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:17.753158 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455784b8-082c-49d2-8835-c9756d1912ee admin admin] PATH_INFO: `/v3/projects/ac90a94a900543c7b7017f2a053a9fda` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:17.755529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455784b8-082c-49d2-8835-c9756d1912ee admin admin] RBAC: Authorizing `identity:delete_project(project_id=ac90a94a900543c7b7017f2a053a9fda)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:17.757455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455784b8-082c-49d2-8835-c9756d1912ee admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:17.758302 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3121a473-2a8c-4d53-b677-756638832605 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:17.759376 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3121a473-2a8c-4d53-b677-756638832605 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:17.759661 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3121a473-2a8c-4d53-b677-756638832605 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:17.760242 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3121a473-2a8c-4d53-b677-756638832605 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:17.760560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3121a473-2a8c-4d53-b677-756638832605 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:17.762921 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3121a473-2a8c-4d53-b677-756638832605 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:17.861878 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1033/2068] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:17 2022] POST /identity/v3/users => generated 293 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:47:17.863047 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3121a473-2a8c-4d53-b677-756638832605 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:17.870516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-455784b8-082c-49d2-8835-c9756d1912ee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ac90a94a900543c7b7017f2a053a9fda', 'request_id': 'req-455784b8-082c-49d2-8835-c9756d1912ee'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:17.874759 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3121a473-2a8c-4d53-b677-756638832605 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:17.876569 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-455784b8-082c-49d2-8835-c9756d1912ee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ac90a94a900543c7b7017f2a053a9fda', 'request_id': 'req-455784b8-082c-49d2-8835-c9756d1912ee'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:17.883630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-455784b8-082c-49d2-8835-c9756d1912ee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ac90a94a900543c7b7017f2a053a9fda', 'request_id': 'req-455784b8-082c-49d2-8835-c9756d1912ee'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:17.925405 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:17.936459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6d881d-7857-4f34-ab59-7feb71ab71e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:17.937107 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6d881d-7857-4f34-ab59-7feb71ab71e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:17.937370 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6d881d-7857-4f34-ab59-7feb71ab71e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:17.937767 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6d881d-7857-4f34-ab59-7feb71ab71e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:17.938405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6d881d-7857-4f34-ab59-7feb71ab71e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:17.940094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6d881d-7857-4f34-ab59-7feb71ab71e3 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:17.953257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1034/2069] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:17 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:47:17.954243 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6d881d-7857-4f34-ab59-7feb71ab71e3 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:17.963995 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6d881d-7857-4f34-ab59-7feb71ab71e3 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:17.969224 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466834c5-7d54-4373-a5e7-304f98ad41f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:17.969787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466834c5-7d54-4373-a5e7-304f98ad41f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:17.970071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466834c5-7d54-4373-a5e7-304f98ad41f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:17.970342 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466834c5-7d54-4373-a5e7-304f98ad41f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:17.971009 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466834c5-7d54-4373-a5e7-304f98ad41f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:17.972593 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466834c5-7d54-4373-a5e7-304f98ad41f0 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:17.974723 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:17.981801 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-455784b8-082c-49d2-8835-c9756d1912ee admin admin] The token cache is being invalidate because project ac90a94a900543c7b7017f2a053a9fda was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:17.982262 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-455784b8-082c-49d2-8835-c9756d1912ee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:17.983622 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-455784b8-082c-49d2-8835-c9756d1912ee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:17.986363 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1035/2070] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:17 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:47:17.986486 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1036/2071] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:17 2022] DELETE /identity/v3/projects/ac90a94a900543c7b7017f2a053a9fda => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:17.988800 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466834c5-7d54-4373-a5e7-304f98ad41f0 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:17.990481 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-466834c5-7d54-4373-a5e7-304f98ad41f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:17.991913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dc6042-44a4-40b8-89a6-844211871494 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:17.992314 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dc6042-44a4-40b8-89a6-844211871494 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:17.992668 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dc6042-44a4-40b8-89a6-844211871494 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:18.003922 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466834c5-7d54-4373-a5e7-304f98ad41f0 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:18.008691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851fd0e1-04ff-4e07-ad3f-57b31026b5c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:18.009498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851fd0e1-04ff-4e07-ad3f-57b31026b5c9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:18.009783 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851fd0e1-04ff-4e07-ad3f-57b31026b5c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:18.010056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851fd0e1-04ff-4e07-ad3f-57b31026b5c9 admin admin] PATH_INFO: `/v3/projects/4f3bb3aa4bba4d61861f179a59de5f13/users/9b3dd405f0d944f89228fe780423b442/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:18.013060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-20dc6042-44a4-40b8-89a6-844211871494 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:18.014304 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-20dc6042-44a4-40b8-89a6-844211871494 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:18.022243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1037/2072] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:47:18.029921 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851fd0e1-04ff-4e07-ad3f-57b31026b5c9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4f3bb3aa4bba4d61861f179a59de5f13, user_id=9b3dd405f0d944f89228fe780423b442, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:18.030377 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f63b3d-b2ec-463d-aa81-1d2bf79d9fb1 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:18.030685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f63b3d-b2ec-463d-aa81-1d2bf79d9fb1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:18.030963 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f63b3d-b2ec-463d-aa81-1d2bf79d9fb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:18.032700 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851fd0e1-04ff-4e07-ad3f-57b31026b5c9 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:18.050224 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e4f63b3d-b2ec-463d-aa81-1d2bf79d9fb1 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:18.051433 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e4f63b3d-b2ec-463d-aa81-1d2bf79d9fb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:18.059177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1038/2073] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:47:18.063316 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f63b3d-b2ec-463d-aa81-1d2bf79d9fb1 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:18.071978 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f63b3d-b2ec-463d-aa81-1d2bf79d9fb1 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:18.077336 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dc9f05-219d-475f-b7be-b5ab2afbe799 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:18.077700 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dc9f05-219d-475f-b7be-b5ab2afbe799 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:18.078200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dc9f05-219d-475f-b7be-b5ab2afbe799 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:18.078700 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dc9f05-219d-475f-b7be-b5ab2afbe799 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:18.079338 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4dc9f05-219d-475f-b7be-b5ab2afbe799 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:47:18.080205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4dc9f05-219d-475f-b7be-b5ab2afbe799 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:18.081901 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4dc9f05-219d-475f-b7be-b5ab2afbe799 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:18.095528 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1039/2074] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:47:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:47:18.099786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dc9f05-219d-475f-b7be-b5ab2afbe799 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:18.109465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1036/2075] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:17 2022] PUT /identity/v3/projects/4f3bb3aa4bba4d61861f179a59de5f13/users/9b3dd405f0d944f89228fe780423b442/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:18.111100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dc9f05-219d-475f-b7be-b5ab2afbe799 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:18.123122 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68707afe-b32e-4f9f-8580-39a44f2346c8 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:18.123436 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68707afe-b32e-4f9f-8580-39a44f2346c8 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:18.123714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68707afe-b32e-4f9f-8580-39a44f2346c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:18.152751 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-68707afe-b32e-4f9f-8580-39a44f2346c8 None None] MFA Rules not processed for user `9b3dd405f0d944f89228fe780423b442`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:18.153928 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-68707afe-b32e-4f9f-8580-39a44f2346c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:18.168166 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f16db63-a435-4b39-8e48-d64bfd42f6a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:18.168776 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f16db63-a435-4b39-8e48-d64bfd42f6a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:18.169054 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f16db63-a435-4b39-8e48-d64bfd42f6a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:18.169328 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f16db63-a435-4b39-8e48-d64bfd42f6a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:18.170069 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f16db63-a435-4b39-8e48-d64bfd42f6a8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:18.172109 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f16db63-a435-4b39-8e48-d64bfd42f6a8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:18.229057 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1037/2076] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:18 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:47:18.233763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68707afe-b32e-4f9f-8580-39a44f2346c8 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:18.244778 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68707afe-b32e-4f9f-8580-39a44f2346c8 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:18.249405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b75064-686d-4826-baac-73e5214f7e32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:18.249995 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b75064-686d-4826-baac-73e5214f7e32 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:18.250277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b75064-686d-4826-baac-73e5214f7e32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:18.250550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b75064-686d-4826-baac-73e5214f7e32 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:18.251235 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b75064-686d-4826-baac-73e5214f7e32 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:18.253144 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b75064-686d-4826-baac-73e5214f7e32 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:18.282652 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1040/2077] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:18 2022] POST /identity/v3/projects => generated 409 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:47:18.286657 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f16db63-a435-4b39-8e48-d64bfd42f6a8 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:18.334405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1038/2078] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:18 2022] POST /identity/v3/projects => generated 387 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:47:18.334618 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:18.335460 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b75064-686d-4826-baac-73e5214f7e32 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:18.351950 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f16db63-a435-4b39-8e48-d64bfd42f6a8 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:18.399094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b75064-686d-4826-baac-73e5214f7e32 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:18.404060 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d07d08-c631-451a-a9e5-bee2ffd4ce21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:18.405287 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d07d08-c631-451a-a9e5-bee2ffd4ce21 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:18.405579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d07d08-c631-451a-a9e5-bee2ffd4ce21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:18.405840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d07d08-c631-451a-a9e5-bee2ffd4ce21 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:18.406536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d07d08-c631-451a-a9e5-bee2ffd4ce21 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:18.408616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d07d08-c631-451a-a9e5-bee2ffd4ce21 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:18.409350 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f37289-afbf-49e8-9a28-c6529b86f99a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:18.409960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f37289-afbf-49e8-9a28-c6529b86f99a admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:18.410241 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f37289-afbf-49e8-9a28-c6529b86f99a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:18.413543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f37289-afbf-49e8-9a28-c6529b86f99a admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:18.413543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f37289-afbf-49e8-9a28-c6529b86f99a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:18.413543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f37289-afbf-49e8-9a28-c6529b86f99a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:18.521833 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1041/2079] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:18 2022] POST /identity/v3/users => generated 305 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:47:18.526055 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1039/2080] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:18 2022] POST /identity/v3/users => generated 300 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:18.526547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f37289-afbf-49e8-9a28-c6529b86f99a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:18.530035 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d07d08-c631-451a-a9e5-bee2ffd4ce21 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:18.540145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f37289-afbf-49e8-9a28-c6529b86f99a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:18.543444 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d07d08-c631-451a-a9e5-bee2ffd4ce21 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:18.546495 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bb6bac-7509-4b3f-a6a8-6dfac8041d00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:18.547380 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bb6bac-7509-4b3f-a6a8-6dfac8041d00 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:18.547662 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bb6bac-7509-4b3f-a6a8-6dfac8041d00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:18.547940 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bb6bac-7509-4b3f-a6a8-6dfac8041d00 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:18.548590 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7bb6bac-7509-4b3f-a6a8-6dfac8041d00 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:18.549538 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c5fef9-8983-48e0-af86-1d1e30e77e9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:18.550206 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c5fef9-8983-48e0-af86-1d1e30e77e9b admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:18.550334 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7bb6bac-7509-4b3f-a6a8-6dfac8041d00 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:18.550488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c5fef9-8983-48e0-af86-1d1e30e77e9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:18.550778 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c5fef9-8983-48e0-af86-1d1e30e77e9b admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:18.551427 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c5fef9-8983-48e0-af86-1d1e30e77e9b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:18.553168 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c5fef9-8983-48e0-af86-1d1e30e77e9b admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:18.564828 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1042/2081] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:18 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:47:18.564999 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1040/2082] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:18 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:47:18.565922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bb6bac-7509-4b3f-a6a8-6dfac8041d00 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:18.571563 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c5fef9-8983-48e0-af86-1d1e30e77e9b admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:18.576981 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bb6bac-7509-4b3f-a6a8-6dfac8041d00 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:18.582762 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c5fef9-8983-48e0-af86-1d1e30e77e9b admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:18.584556 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331cd58a-e809-4634-a757-44f6a172a121 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:18.584556 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331cd58a-e809-4634-a757-44f6a172a121 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:18.584556 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331cd58a-e809-4634-a757-44f6a172a121 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:18.584803 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331cd58a-e809-4634-a757-44f6a172a121 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:18.585492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331cd58a-e809-4634-a757-44f6a172a121 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:18.587125 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331cd58a-e809-4634-a757-44f6a172a121 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:18.588231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa6d731-7b06-45e9-b4c3-56566f0207cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:18.588954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa6d731-7b06-45e9-b4c3-56566f0207cf admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:18.590766 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa6d731-7b06-45e9-b4c3-56566f0207cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:18.590766 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa6d731-7b06-45e9-b4c3-56566f0207cf admin admin] PATH_INFO: `/v3/projects/59079381c1824e989b2ee73803e4f0f7/users/7785772590ce481ea8b2435d6ec145d8/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:18.600226 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1043/2083] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:18 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:47:18.604976 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331cd58a-e809-4634-a757-44f6a172a121 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:18.606109 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa6d731-7b06-45e9-b4c3-56566f0207cf admin admin] RBAC: Authorizing `identity:create_grant(project_id=59079381c1824e989b2ee73803e4f0f7, user_id=7785772590ce481ea8b2435d6ec145d8, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:18.609353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa6d731-7b06-45e9-b4c3-56566f0207cf admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:18.618082 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331cd58a-e809-4634-a757-44f6a172a121 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:18.623763 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f02ecce-7c5a-4839-a3bc-13e79a12e738 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:18.624431 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f02ecce-7c5a-4839-a3bc-13e79a12e738 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:18.624738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f02ecce-7c5a-4839-a3bc-13e79a12e738 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:18.625004 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f02ecce-7c5a-4839-a3bc-13e79a12e738 admin admin] PATH_INFO: `/v3/projects/584de4573964403fb0bace83b40e0761/users/ed9d2408c78d426d8cbfcfd7839da69b/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:18.633733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:18.644247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f02ecce-7c5a-4839-a3bc-13e79a12e738 admin admin] RBAC: Authorizing `identity:create_grant(project_id=584de4573964403fb0bace83b40e0761, user_id=ed9d2408c78d426d8cbfcfd7839da69b, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:18.647017 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f02ecce-7c5a-4839-a3bc-13e79a12e738 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:18.687005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1041/2084] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:18 2022] PUT /identity/v3/projects/59079381c1824e989b2ee73803e4f0f7/users/7785772590ce481ea8b2435d6ec145d8/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:18.691132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa6d731-7b06-45e9-b4c3-56566f0207cf admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:18.710196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1044/2085] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:18 2022] PUT /identity/v3/projects/584de4573964403fb0bace83b40e0761/users/ed9d2408c78d426d8cbfcfd7839da69b/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:18.726332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf0e5cb-cf66-41e2-a24d-fd16666fd620 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:18.726647 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf0e5cb-cf66-41e2-a24d-fd16666fd620 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:18.727088 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf0e5cb-cf66-41e2-a24d-fd16666fd620 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:18.755787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7bf0e5cb-cf66-41e2-a24d-fd16666fd620 None None] MFA Rules not processed for user `ed9d2408c78d426d8cbfcfd7839da69b`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:18.756994 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7bf0e5cb-cf66-41e2-a24d-fd16666fd620 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:18.765723 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa6d731-7b06-45e9-b4c3-56566f0207cf admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:18.836475 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f95c34-c603-4bf7-aa3c-1a64c2a9cd23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:18.837270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f95c34-c603-4bf7-aa3c-1a64c2a9cd23 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:18.837612 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f95c34-c603-4bf7-aa3c-1a64c2a9cd23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:18.837894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f95c34-c603-4bf7-aa3c-1a64c2a9cd23 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:18.838578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f95c34-c603-4bf7-aa3c-1a64c2a9cd23 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:18.840122 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1045/2086] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:18 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:47:18.840237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f95c34-c603-4bf7-aa3c-1a64c2a9cd23 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:18.846451 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf0e5cb-cf66-41e2-a24d-fd16666fd620 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:18.858425 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1042/2087] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:18 2022] GET /identity/v3/roles => generated 1420 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:47:18.859344 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f95c34-c603-4bf7-aa3c-1a64c2a9cd23 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:18.860380 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf0e5cb-cf66-41e2-a24d-fd16666fd620 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:18.866778 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a6ef38-0423-4f5f-b3d8-487d20e9716a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:18.871205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a6ef38-0423-4f5f-b3d8-487d20e9716a admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:18.871205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a6ef38-0423-4f5f-b3d8-487d20e9716a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:18.871205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a6ef38-0423-4f5f-b3d8-487d20e9716a admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:18.871205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a6ef38-0423-4f5f-b3d8-487d20e9716a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:18.872239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f95c34-c603-4bf7-aa3c-1a64c2a9cd23 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:18.872396 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a6ef38-0423-4f5f-b3d8-487d20e9716a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:18.878274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85bd3ff8-1565-49ef-a5aa-9e548e902004 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:18.879117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bd3ff8-1565-49ef-a5aa-9e548e902004 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:18.879469 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bd3ff8-1565-49ef-a5aa-9e548e902004 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:18.879809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bd3ff8-1565-49ef-a5aa-9e548e902004 admin admin] PATH_INFO: `/v3/projects/59079381c1824e989b2ee73803e4f0f7/users/7785772590ce481ea8b2435d6ec145d8/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:18.881853 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85bd3ff8-1565-49ef-a5aa-9e548e902004 admin admin] RBAC: Authorizing `identity:create_grant(project_id=59079381c1824e989b2ee73803e4f0f7, user_id=7785772590ce481ea8b2435d6ec145d8, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:18.884826 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85bd3ff8-1565-49ef-a5aa-9e548e902004 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:18.895978 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1043/2088] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:18 2022] PUT /identity/v3/projects/59079381c1824e989b2ee73803e4f0f7/users/7785772590ce481ea8b2435d6ec145d8/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:18.908426 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b1fcb5-b593-4b27-9d6f-6fefd13f7ac2 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:18.908831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b1fcb5-b593-4b27-9d6f-6fefd13f7ac2 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:18.909121 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b1fcb5-b593-4b27-9d6f-6fefd13f7ac2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:18.941107 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-12b1fcb5-b593-4b27-9d6f-6fefd13f7ac2 None None] MFA Rules not processed for user `7785772590ce481ea8b2435d6ec145d8`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:18.942388 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-12b1fcb5-b593-4b27-9d6f-6fefd13f7ac2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:18.986002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1046/2089] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:18 2022] POST /identity/v3/projects => generated 405 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:47:18.995141 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a6ef38-0423-4f5f-b3d8-487d20e9716a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:18.997453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-02a6ef38-0423-4f5f-b3d8-487d20e9716a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:19.031068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1044/2090] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:18 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:47:19.038284 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b1fcb5-b593-4b27-9d6f-6fefd13f7ac2 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:19.081656 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a6ef38-0423-4f5f-b3d8-487d20e9716a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:19.086722 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19abd747-6a54-4cd5-b5bc-ecad4bc54f9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:19.090624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19abd747-6a54-4cd5-b5bc-ecad4bc54f9f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:19.090624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19abd747-6a54-4cd5-b5bc-ecad4bc54f9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:19.090624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19abd747-6a54-4cd5-b5bc-ecad4bc54f9f admin admin] PATH_INFO: `/v3/users/2087cc96a15646528ee3c5d14443bef6` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:19.091961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19abd747-6a54-4cd5-b5bc-ecad4bc54f9f admin admin] RBAC: Authorizing `identity:delete_user(user_id=2087cc96a15646528ee3c5d14443bef6)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:19.095558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19abd747-6a54-4cd5-b5bc-ecad4bc54f9f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:19.123243 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b1fcb5-b593-4b27-9d6f-6fefd13f7ac2 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:19.130407 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd5a8aa-da73-4460-9c5b-71702ed2992e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:19.131070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd5a8aa-da73-4460-9c5b-71702ed2992e admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:19.131353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd5a8aa-da73-4460-9c5b-71702ed2992e admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:19.131630 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd5a8aa-da73-4460-9c5b-71702ed2992e admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:19.132267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd5a8aa-da73-4460-9c5b-71702ed2992e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:19.134446 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd5a8aa-da73-4460-9c5b-71702ed2992e admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:19.177892 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:19.237762 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:19.244846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1045/2091] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:19 2022] POST /identity/v3/users => generated 309 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:19.245868 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd5a8aa-da73-4460-9c5b-71702ed2992e admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:19.247825 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-cbd5a8aa-da73-4460-9c5b-71702ed2992e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:19.318502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd5a8aa-da73-4460-9c5b-71702ed2992e admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:19.323384 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87b4cb3-96c1-414b-8444-d848aa971d5d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:19.324003 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87b4cb3-96c1-414b-8444-d848aa971d5d service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:19.324439 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87b4cb3-96c1-414b-8444-d848aa971d5d service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:19.324830 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87b4cb3-96c1-414b-8444-d848aa971d5d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:19.326262 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-19abd747-6a54-4cd5-b5bc-ecad4bc54f9f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2087cc96a15646528ee3c5d14443bef6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:19.327143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-19abd747-6a54-4cd5-b5bc-ecad4bc54f9f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2087cc96a15646528ee3c5d14443bef6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:19.327834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-19abd747-6a54-4cd5-b5bc-ecad4bc54f9f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2087cc96a15646528ee3c5d14443bef6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:19.337441 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-19abd747-6a54-4cd5-b5bc-ecad4bc54f9f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2087cc96a15646528ee3c5d14443bef6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:19.412676 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-19abd747-6a54-4cd5-b5bc-ecad4bc54f9f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2087cc96a15646528ee3c5d14443bef6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:19.419111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87b4cb3-96c1-414b-8444-d848aa971d5d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:19.421422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87b4cb3-96c1-414b-8444-d848aa971d5d service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:19.433775 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1046/2092] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:19 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:19.434761 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87b4cb3-96c1-414b-8444-d848aa971d5d service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:19.450383 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87b4cb3-96c1-414b-8444-d848aa971d5d service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:19.463854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1932b6eb-7766-4ea3-b050-556517104c3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:19.467383 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1932b6eb-7766-4ea3-b050-556517104c3f admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:19.467383 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1932b6eb-7766-4ea3-b050-556517104c3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:19.467383 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1932b6eb-7766-4ea3-b050-556517104c3f admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:19.467383 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1932b6eb-7766-4ea3-b050-556517104c3f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:19.469281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1047/2093] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:18 2022] DELETE /identity/v3/users/2087cc96a15646528ee3c5d14443bef6 => generated 0 bytes in 483 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:19.470318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1932b6eb-7766-4ea3-b050-556517104c3f admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:19.475865 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccf3b14-955e-479a-bfb7-4f01759142d3 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:19.476434 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccf3b14-955e-479a-bfb7-4f01759142d3 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:19.476930 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccf3b14-955e-479a-bfb7-4f01759142d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:19.481901 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1047/2094] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:19 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:19.489790 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1932b6eb-7766-4ea3-b050-556517104c3f admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:19.489973 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-1932b6eb-7766-4ea3-b050-556517104c3f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:19.504267 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-eccf3b14-955e-479a-bfb7-4f01759142d3 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:19.505623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-eccf3b14-955e-479a-bfb7-4f01759142d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:19.508319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1932b6eb-7766-4ea3-b050-556517104c3f admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:19.572362 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1048/2095] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:47:19.573584 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa51aeb-95ed-4c8d-8cd6-a9f7d351eb84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:19.574259 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa51aeb-95ed-4c8d-8cd6-a9f7d351eb84 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:19.574692 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa51aeb-95ed-4c8d-8cd6-a9f7d351eb84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:19.574987 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa51aeb-95ed-4c8d-8cd6-a9f7d351eb84 admin admin] PATH_INFO: `/v3/projects/aba854709eb547f9b7998ff458a00168/users/c9ea2c12e28a4df8b4335522cc60b21d/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:19.586889 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccf3b14-955e-479a-bfb7-4f01759142d3 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:19.591369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-eccf3b14-955e-479a-bfb7-4f01759142d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:19.599797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa51aeb-95ed-4c8d-8cd6-a9f7d351eb84 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aba854709eb547f9b7998ff458a00168, user_id=c9ea2c12e28a4df8b4335522cc60b21d, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:19.599797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa51aeb-95ed-4c8d-8cd6-a9f7d351eb84 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:19.630394 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:19.676335 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccf3b14-955e-479a-bfb7-4f01759142d3 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:19.680091 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1048/2096] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:19 2022] PUT /identity/v3/projects/aba854709eb547f9b7998ff458a00168/users/c9ea2c12e28a4df8b4335522cc60b21d/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:47:19.686340 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa51aeb-95ed-4c8d-8cd6-a9f7d351eb84 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:19.753567 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033a56d0-64c4-49b0-9bf6-41a486dfa34d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:19.754219 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033a56d0-64c4-49b0-9bf6-41a486dfa34d service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:19.754507 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033a56d0-64c4-49b0-9bf6-41a486dfa34d service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:19.754817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033a56d0-64c4-49b0-9bf6-41a486dfa34d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:19.779433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa51aeb-95ed-4c8d-8cd6-a9f7d351eb84 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:19.789395 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643d1878-f95e-4a7c-b232-9fc814e1c0d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:19.789395 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643d1878-f95e-4a7c-b232-9fc814e1c0d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:19.789395 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643d1878-f95e-4a7c-b232-9fc814e1c0d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:19.789395 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643d1878-f95e-4a7c-b232-9fc814e1c0d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:19.789837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033a56d0-64c4-49b0-9bf6-41a486dfa34d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:19.790358 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643d1878-f95e-4a7c-b232-9fc814e1c0d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:19.791254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033a56d0-64c4-49b0-9bf6-41a486dfa34d service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:19.792733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643d1878-f95e-4a7c-b232-9fc814e1c0d4 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:19.809288 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1049/2097] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:19 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:19.810572 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643d1878-f95e-4a7c-b232-9fc814e1c0d4 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:19.811680 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1049/2098] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:19.822508 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643d1878-f95e-4a7c-b232-9fc814e1c0d4 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:19.835877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab96a70e-5f3d-4a2e-91ce-830844e65cef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:19.840739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab96a70e-5f3d-4a2e-91ce-830844e65cef admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:19.840739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab96a70e-5f3d-4a2e-91ce-830844e65cef admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:19.840739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab96a70e-5f3d-4a2e-91ce-830844e65cef admin admin] PATH_INFO: `/v3/projects/aba854709eb547f9b7998ff458a00168/users/c9ea2c12e28a4df8b4335522cc60b21d/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:19.840739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab96a70e-5f3d-4a2e-91ce-830844e65cef admin admin] RBAC: Authorizing `identity:create_grant(project_id=aba854709eb547f9b7998ff458a00168, user_id=c9ea2c12e28a4df8b4335522cc60b21d, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:19.842783 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab96a70e-5f3d-4a2e-91ce-830844e65cef admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:19.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:19.862764 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1050/2099] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:19 2022] PUT /identity/v3/projects/aba854709eb547f9b7998ff458a00168/users/c9ea2c12e28a4df8b4335522cc60b21d/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:19.868757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddc967c-4ca1-4d99-801c-dac84322ddad None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:19.869043 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddc967c-4ca1-4d99-801c-dac84322ddad None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:19.869339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddc967c-4ca1-4d99-801c-dac84322ddad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:19.925183 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-8ddc967c-4ca1-4d99-801c-dac84322ddad None None] MFA Rules not processed for user `c9ea2c12e28a4df8b4335522cc60b21d`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:19.925183 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8ddc967c-4ca1-4d99-801c-dac84322ddad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:20.029043 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1050/2100] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:19 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:20.039126 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab96a70e-5f3d-4a2e-91ce-830844e65cef admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:20.040904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ab96a70e-5f3d-4a2e-91ce-830844e65cef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:20.126956 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab96a70e-5f3d-4a2e-91ce-830844e65cef admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:20.142546 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84463e5c-434d-445d-9cc0-77a4458dd498 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:20.148061 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84463e5c-434d-445d-9cc0-77a4458dd498 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:20.148061 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84463e5c-434d-445d-9cc0-77a4458dd498 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:20.148367 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84463e5c-434d-445d-9cc0-77a4458dd498 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:20.166896 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84463e5c-434d-445d-9cc0-77a4458dd498 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:20.169365 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84463e5c-434d-445d-9cc0-77a4458dd498 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:20.184634 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1051/2101] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:20 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:20.395371 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddc967c-4ca1-4d99-801c-dac84322ddad None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:20.396914 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8ddc967c-4ca1-4d99-801c-dac84322ddad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:20.490994 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddc967c-4ca1-4d99-801c-dac84322ddad None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:20.498421 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a639b6c5-2f10-4f50-a3f9-b631a1c60b55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:20.499458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a639b6c5-2f10-4f50-a3f9-b631a1c60b55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:20.499953 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a639b6c5-2f10-4f50-a3f9-b631a1c60b55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:20.500453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a639b6c5-2f10-4f50-a3f9-b631a1c60b55 admin admin] PATH_INFO: `/v3/projects/82de32938a7340b391a698ddf94c8390` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:20.501815 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a639b6c5-2f10-4f50-a3f9-b631a1c60b55 admin admin] RBAC: Authorizing `identity:delete_project(project_id=82de32938a7340b391a698ddf94c8390)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:20.504202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a639b6c5-2f10-4f50-a3f9-b631a1c60b55 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:20.605665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84463e5c-434d-445d-9cc0-77a4458dd498 service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:20.614026 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84463e5c-434d-445d-9cc0-77a4458dd498 service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:20.618547 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cbb6e8-2065-4e62-bbe8-b17025edbf80 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:20.619287 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cbb6e8-2065-4e62-bbe8-b17025edbf80 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:20.619912 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cbb6e8-2065-4e62-bbe8-b17025edbf80 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:20.620193 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cbb6e8-2065-4e62-bbe8-b17025edbf80 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:20.622114 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-38cbb6e8-2065-4e62-bbe8-b17025edbf80 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:20.672711 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a639b6c5-2f10-4f50-a3f9-b631a1c60b55 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '82de32938a7340b391a698ddf94c8390', 'request_id': 'req-a639b6c5-2f10-4f50-a3f9-b631a1c60b55'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:20.682286 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a639b6c5-2f10-4f50-a3f9-b631a1c60b55 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '82de32938a7340b391a698ddf94c8390', 'request_id': 'req-a639b6c5-2f10-4f50-a3f9-b631a1c60b55'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:20.690690 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a639b6c5-2f10-4f50-a3f9-b631a1c60b55 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '82de32938a7340b391a698ddf94c8390', 'request_id': 'req-a639b6c5-2f10-4f50-a3f9-b631a1c60b55'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:20.708680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cbb6e8-2065-4e62-bbe8-b17025edbf80 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:20.717373 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cbb6e8-2065-4e62-bbe8-b17025edbf80 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:20.735599 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1052/2102] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:20 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:47:20.775320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a639b6c5-2f10-4f50-a3f9-b631a1c60b55 admin admin] The token cache is being invalidate because project 82de32938a7340b391a698ddf94c8390 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:20.775716 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a639b6c5-2f10-4f50-a3f9-b631a1c60b55 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:20.776454 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a639b6c5-2f10-4f50-a3f9-b631a1c60b55 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:20.781543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1051/2103] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:20 2022] DELETE /identity/v3/projects/82de32938a7340b391a698ddf94c8390 => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:20.783144 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cbb6e8-2065-4e62-bbe8-b17025edbf80 service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:20.796007 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cbb6e8-2065-4e62-bbe8-b17025edbf80 service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:20.873472 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e57efd-d774-402d-b204-a92c18e9dc95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:20.874740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e57efd-d774-402d-b204-a92c18e9dc95 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:20.888646 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e57efd-d774-402d-b204-a92c18e9dc95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:20.888646 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e57efd-d774-402d-b204-a92c18e9dc95 admin admin] PATH_INFO: `/v3/users/4d73b57c74c44615bc34b59348b6e1e7` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:20.888646 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e57efd-d774-402d-b204-a92c18e9dc95 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4d73b57c74c44615bc34b59348b6e1e7)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:20.888646 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e57efd-d774-402d-b204-a92c18e9dc95 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:20.906446 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a639b6c5-2f10-4f50-a3f9-b631a1c60b55 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:20.908338 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a639b6c5-2f10-4f50-a3f9-b631a1c60b55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:20.922726 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a639b6c5-2f10-4f50-a3f9-b631a1c60b55 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:20.929555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6503b7-4843-4197-b057-d696cf5611af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:20.930214 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6503b7-4843-4197-b057-d696cf5611af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:20.930497 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6503b7-4843-4197-b057-d696cf5611af admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:20.930801 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6503b7-4843-4197-b057-d696cf5611af admin admin] PATH_INFO: `/v3/users/9b3dd405f0d944f89228fe780423b442` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:20.931978 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6503b7-4843-4197-b057-d696cf5611af admin admin] RBAC: Authorizing `identity:delete_user(user_id=9b3dd405f0d944f89228fe780423b442)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:20.937272 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6503b7-4843-4197-b057-d696cf5611af admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:21.011935 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:21.142756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-84e57efd-d774-402d-b204-a92c18e9dc95 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4d73b57c74c44615bc34b59348b6e1e7'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:21.170192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-84e57efd-d774-402d-b204-a92c18e9dc95 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4d73b57c74c44615bc34b59348b6e1e7'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:21.171358 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-84e57efd-d774-402d-b204-a92c18e9dc95 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4d73b57c74c44615bc34b59348b6e1e7'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:21.181466 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ce6503b7-4843-4197-b057-d696cf5611af admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9b3dd405f0d944f89228fe780423b442'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:21.182152 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ce6503b7-4843-4197-b057-d696cf5611af admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9b3dd405f0d944f89228fe780423b442'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:21.182839 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ce6503b7-4843-4197-b057-d696cf5611af admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9b3dd405f0d944f89228fe780423b442'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:21.189832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:21.193535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ce6503b7-4843-4197-b057-d696cf5611af admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9b3dd405f0d944f89228fe780423b442'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:21.275286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-84e57efd-d774-402d-b204-a92c18e9dc95 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4d73b57c74c44615bc34b59348b6e1e7'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:21.286732 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-84e57efd-d774-402d-b204-a92c18e9dc95 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4d73b57c74c44615bc34b59348b6e1e7'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:21.298431 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1053/2104] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:20 2022] DELETE /identity/v3/users/4d73b57c74c44615bc34b59348b6e1e7 => generated 0 bytes in 516 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:21.467045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ce6503b7-4843-4197-b057-d696cf5611af admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9b3dd405f0d944f89228fe780423b442'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:21.488309 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1052/2105] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:20 2022] DELETE /identity/v3/users/9b3dd405f0d944f89228fe780423b442 => generated 0 bytes in 582 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:21.492839 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc6becf-465a-42a6-bed6-41d64df5aed7 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:21.493126 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc6becf-465a-42a6-bed6-41d64df5aed7 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:21.493402 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc6becf-465a-42a6-bed6-41d64df5aed7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:21.525210 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-ddc6becf-465a-42a6-bed6-41d64df5aed7 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:21.531700 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ddc6becf-465a-42a6-bed6-41d64df5aed7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:21.666127 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1054/2106] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:21.688781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6503b7-4843-4197-b057-d696cf5611af admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:21.690653 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ce6503b7-4843-4197-b057-d696cf5611af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:21.808869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6503b7-4843-4197-b057-d696cf5611af admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:21.815220 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e081ad-23e5-48b2-986e-7ac2ebe3eae4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:21.816005 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e081ad-23e5-48b2-986e-7ac2ebe3eae4 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:21.816252 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e081ad-23e5-48b2-986e-7ac2ebe3eae4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:21.816704 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e081ad-23e5-48b2-986e-7ac2ebe3eae4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:21.829491 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e081ad-23e5-48b2-986e-7ac2ebe3eae4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:21.830795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc6becf-465a-42a6-bed6-41d64df5aed7 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:21.831033 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e081ad-23e5-48b2-986e-7ac2ebe3eae4 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:21.833060 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ddc6becf-465a-42a6-bed6-41d64df5aed7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:21.848309 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1053/2107] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:21.855128 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc6becf-465a-42a6-bed6-41d64df5aed7 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:21.880865 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbb9b1a-c882-463c-9fd0-165fd9f3b4cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:21.881675 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbb9b1a-c882-463c-9fd0-165fd9f3b4cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:21.881998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbb9b1a-c882-463c-9fd0-165fd9f3b4cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:21.882269 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbb9b1a-c882-463c-9fd0-165fd9f3b4cc admin admin] PATH_INFO: `/v3/projects/a78c4b96aca44d8aa8cf6be70b245120` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:21.884923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbb9b1a-c882-463c-9fd0-165fd9f3b4cc admin admin] RBAC: Authorizing `identity:delete_project(project_id=a78c4b96aca44d8aa8cf6be70b245120)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:21.889455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbb9b1a-c882-463c-9fd0-165fd9f3b4cc admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:22.104176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-2dbb9b1a-c882-463c-9fd0-165fd9f3b4cc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a78c4b96aca44d8aa8cf6be70b245120', 'request_id': 'req-2dbb9b1a-c882-463c-9fd0-165fd9f3b4cc'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:22.112345 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-2dbb9b1a-c882-463c-9fd0-165fd9f3b4cc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a78c4b96aca44d8aa8cf6be70b245120', 'request_id': 'req-2dbb9b1a-c882-463c-9fd0-165fd9f3b4cc'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:22.120052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-2dbb9b1a-c882-463c-9fd0-165fd9f3b4cc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a78c4b96aca44d8aa8cf6be70b245120', 'request_id': 'req-2dbb9b1a-c882-463c-9fd0-165fd9f3b4cc'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:22.206977 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-2dbb9b1a-c882-463c-9fd0-165fd9f3b4cc admin admin] The token cache is being invalidate because project a78c4b96aca44d8aa8cf6be70b245120 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:22.208071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-2dbb9b1a-c882-463c-9fd0-165fd9f3b4cc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:22.208757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-2dbb9b1a-c882-463c-9fd0-165fd9f3b4cc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:22.210541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1055/2108] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:21 2022] DELETE /identity/v3/projects/a78c4b96aca44d8aa8cf6be70b245120 => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:22.214224 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e081ad-23e5-48b2-986e-7ac2ebe3eae4 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:22.226331 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e081ad-23e5-48b2-986e-7ac2ebe3eae4 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:22.232232 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:22.296617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f181082-392c-4a03-a474-08dac561d2f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:22.297370 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f181082-392c-4a03-a474-08dac561d2f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:22.297720 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f181082-392c-4a03-a474-08dac561d2f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:22.297999 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f181082-392c-4a03-a474-08dac561d2f7 admin admin] PATH_INFO: `/v3/users/1ec537f4d1a64a4e9330d3cec97e513e` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:22.299147 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f181082-392c-4a03-a474-08dac561d2f7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1ec537f4d1a64a4e9330d3cec97e513e)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:22.301087 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f181082-392c-4a03-a474-08dac561d2f7 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:22.494198 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:22.514139 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8f181082-392c-4a03-a474-08dac561d2f7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1ec537f4d1a64a4e9330d3cec97e513e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:22.519585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8f181082-392c-4a03-a474-08dac561d2f7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1ec537f4d1a64a4e9330d3cec97e513e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:22.523830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8f181082-392c-4a03-a474-08dac561d2f7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1ec537f4d1a64a4e9330d3cec97e513e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:22.540470 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8f181082-392c-4a03-a474-08dac561d2f7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1ec537f4d1a64a4e9330d3cec97e513e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:22.590492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbb9b1a-c882-463c-9fd0-165fd9f3b4cc admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:22.592042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-2dbb9b1a-c882-463c-9fd0-165fd9f3b4cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:22.607063 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8f181082-392c-4a03-a474-08dac561d2f7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1ec537f4d1a64a4e9330d3cec97e513e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:22.616501 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbb9b1a-c882-463c-9fd0-165fd9f3b4cc admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:22.635651 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dad49be-7485-40c9-963c-e6175a266b33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:22.636394 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dad49be-7485-40c9-963c-e6175a266b33 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:22.636798 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dad49be-7485-40c9-963c-e6175a266b33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:22.637139 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dad49be-7485-40c9-963c-e6175a266b33 admin admin] PATH_INFO: `/v3/users/bc75456a74bd456698580f490ad051f0` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:22.639045 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dad49be-7485-40c9-963c-e6175a266b33 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bc75456a74bd456698580f490ad051f0)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:22.641258 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dad49be-7485-40c9-963c-e6175a266b33 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:22.656853 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1054/2109] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:22 2022] DELETE /identity/v3/users/1ec537f4d1a64a4e9330d3cec97e513e => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:47:22.856621 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0dad49be-7485-40c9-963c-e6175a266b33 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bc75456a74bd456698580f490ad051f0'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:22.871219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0dad49be-7485-40c9-963c-e6175a266b33 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bc75456a74bd456698580f490ad051f0'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:22.871940 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0dad49be-7485-40c9-963c-e6175a266b33 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bc75456a74bd456698580f490ad051f0'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:22.952606 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f181082-392c-4a03-a474-08dac561d2f7 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:22.954276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8f181082-392c-4a03-a474-08dac561d2f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:23.006143 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0dad49be-7485-40c9-963c-e6175a266b33 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bc75456a74bd456698580f490ad051f0'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:23.016329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0dad49be-7485-40c9-963c-e6175a266b33 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bc75456a74bd456698580f490ad051f0'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:23.026037 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1056/2110] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:22 2022] DELETE /identity/v3/users/bc75456a74bd456698580f490ad051f0 => generated 0 bytes in 436 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:23.043797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be28210-c80b-4de5-9614-ec07b017be18 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:23.044427 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be28210-c80b-4de5-9614-ec07b017be18 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:23.045017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be28210-c80b-4de5-9614-ec07b017be18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:23.062672 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f181082-392c-4a03-a474-08dac561d2f7 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:23.089464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-8be28210-c80b-4de5-9614-ec07b017be18 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:23.090458 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-8be28210-c80b-4de5-9614-ec07b017be18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:23.138639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc706e0-ab8d-486e-8348-e8d2bf99a1d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:23.139520 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc706e0-ab8d-486e-8348-e8d2bf99a1d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:23.139957 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc706e0-ab8d-486e-8348-e8d2bf99a1d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:23.140415 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc706e0-ab8d-486e-8348-e8d2bf99a1d9 admin admin] PATH_INFO: `/v3/projects/4f3bb3aa4bba4d61861f179a59de5f13` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:23.142065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc706e0-ab8d-486e-8348-e8d2bf99a1d9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4f3bb3aa4bba4d61861f179a59de5f13)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:23.144036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc706e0-ab8d-486e-8348-e8d2bf99a1d9 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:23.172063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1057/2111] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:23.182134 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be28210-c80b-4de5-9614-ec07b017be18 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:23.183639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-8be28210-c80b-4de5-9614-ec07b017be18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:23.260253 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be28210-c80b-4de5-9614-ec07b017be18 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:23.267913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0cc706e0-ab8d-486e-8348-e8d2bf99a1d9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4f3bb3aa4bba4d61861f179a59de5f13', 'request_id': 'req-0cc706e0-ab8d-486e-8348-e8d2bf99a1d9'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:23.275992 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0cc706e0-ab8d-486e-8348-e8d2bf99a1d9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4f3bb3aa4bba4d61861f179a59de5f13', 'request_id': 'req-0cc706e0-ab8d-486e-8348-e8d2bf99a1d9'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:23.283861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0cc706e0-ab8d-486e-8348-e8d2bf99a1d9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4f3bb3aa4bba4d61861f179a59de5f13', 'request_id': 'req-0cc706e0-ab8d-486e-8348-e8d2bf99a1d9'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:23.317563 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1d0ecc-23da-49d8-ab77-4c30af168353 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:23.318108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1d0ecc-23da-49d8-ab77-4c30af168353 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:23.318385 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1d0ecc-23da-49d8-ab77-4c30af168353 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:23.318688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1d0ecc-23da-49d8-ab77-4c30af168353 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:23.396522 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1d0ecc-23da-49d8-ab77-4c30af168353 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:23.396522 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1d0ecc-23da-49d8-ab77-4c30af168353 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:23.399696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0cc706e0-ab8d-486e-8348-e8d2bf99a1d9 admin admin] The token cache is being invalidate because project 4f3bb3aa4bba4d61861f179a59de5f13 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:23.400912 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0cc706e0-ab8d-486e-8348-e8d2bf99a1d9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:23.402646 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0cc706e0-ab8d-486e-8348-e8d2bf99a1d9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:23.408849 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1055/2112] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:22 2022] DELETE /identity/v3/projects/4f3bb3aa4bba4d61861f179a59de5f13 => generated 0 bytes in 457 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:23.410327 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1058/2113] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:47:23.412600 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1d0ecc-23da-49d8-ab77-4c30af168353 service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:23.414297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-1b1d0ecc-23da-49d8-ab77-4c30af168353 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:23.425596 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc706e0-ab8d-486e-8348-e8d2bf99a1d9 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:23.431657 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0cc706e0-ab8d-486e-8348-e8d2bf99a1d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:23.435563 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1d0ecc-23da-49d8-ab77-4c30af168353 service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:23.446513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5717ab-a855-410e-920f-478c60a788c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:23.448576 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5717ab-a855-410e-920f-478c60a788c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:23.448864 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5717ab-a855-410e-920f-478c60a788c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:23.449138 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5717ab-a855-410e-920f-478c60a788c0 admin admin] PATH_INFO: `/v3/users/7785772590ce481ea8b2435d6ec145d8` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:23.450440 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5717ab-a855-410e-920f-478c60a788c0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7785772590ce481ea8b2435d6ec145d8)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:23.454431 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5717ab-a855-410e-920f-478c60a788c0 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:23.459187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc706e0-ab8d-486e-8348-e8d2bf99a1d9 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:23.477832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38cccb9-908c-40ee-a07e-83e8d55fb4b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:23.479811 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38cccb9-908c-40ee-a07e-83e8d55fb4b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:23.480203 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38cccb9-908c-40ee-a07e-83e8d55fb4b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:23.480490 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38cccb9-908c-40ee-a07e-83e8d55fb4b4 admin admin] PATH_INFO: `/v3/projects/d951df5495244097a4026510af3b616f` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:23.485094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38cccb9-908c-40ee-a07e-83e8d55fb4b4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d951df5495244097a4026510af3b616f)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:23.488234 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38cccb9-908c-40ee-a07e-83e8d55fb4b4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:23.668794 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-ed5717ab-a855-410e-920f-478c60a788c0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7785772590ce481ea8b2435d6ec145d8'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:23.693724 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-ed5717ab-a855-410e-920f-478c60a788c0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7785772590ce481ea8b2435d6ec145d8'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:23.695483 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-ed5717ab-a855-410e-920f-478c60a788c0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7785772590ce481ea8b2435d6ec145d8'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:23.704735 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d38cccb9-908c-40ee-a07e-83e8d55fb4b4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd951df5495244097a4026510af3b616f', 'request_id': 'req-d38cccb9-908c-40ee-a07e-83e8d55fb4b4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:23.710828 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d38cccb9-908c-40ee-a07e-83e8d55fb4b4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd951df5495244097a4026510af3b616f', 'request_id': 'req-d38cccb9-908c-40ee-a07e-83e8d55fb4b4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:23.719828 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d38cccb9-908c-40ee-a07e-83e8d55fb4b4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd951df5495244097a4026510af3b616f', 'request_id': 'req-d38cccb9-908c-40ee-a07e-83e8d55fb4b4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:23.755849 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-ed5717ab-a855-410e-920f-478c60a788c0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7785772590ce481ea8b2435d6ec145d8'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:23.767815 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-ed5717ab-a855-410e-920f-478c60a788c0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7785772590ce481ea8b2435d6ec145d8'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:23.775545 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1059/2114] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:23 2022] DELETE /identity/v3/users/7785772590ce481ea8b2435d6ec145d8 => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:47:23.878167 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d38cccb9-908c-40ee-a07e-83e8d55fb4b4 admin admin] The token cache is being invalidate because project d951df5495244097a4026510af3b616f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:23.878814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d38cccb9-908c-40ee-a07e-83e8d55fb4b4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:23.879881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d38cccb9-908c-40ee-a07e-83e8d55fb4b4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:23.881971 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1056/2115] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:23 2022] DELETE /identity/v3/projects/d951df5495244097a4026510af3b616f => generated 0 bytes in 473 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:23.887197 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4a7c08-3464-43bb-91b2-1ea045f64f63 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:23.887418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4a7c08-3464-43bb-91b2-1ea045f64f63 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:23.887752 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4a7c08-3464-43bb-91b2-1ea045f64f63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:23.911056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-ea4a7c08-3464-43bb-91b2-1ea045f64f63 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:23.912448 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ea4a7c08-3464-43bb-91b2-1ea045f64f63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:23.995826 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1060/2116] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:24.004096 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9bcc70-4f99-4979-b36c-f8e54894f488 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:24.004388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9bcc70-4f99-4979-b36c-f8e54894f488 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:24.004670 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9bcc70-4f99-4979-b36c-f8e54894f488 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:24.028015 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-2e9bcc70-4f99-4979-b36c-f8e54894f488 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:24.029276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2e9bcc70-4f99-4979-b36c-f8e54894f488 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:24.040575 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1057/2117] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:24.045045 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4a7c08-3464-43bb-91b2-1ea045f64f63 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:24.060077 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4a7c08-3464-43bb-91b2-1ea045f64f63 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:24.067701 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26f4f5f-7a45-480e-804b-95dc01200ff9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:24.068443 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26f4f5f-7a45-480e-804b-95dc01200ff9 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:24.068803 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26f4f5f-7a45-480e-804b-95dc01200ff9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:24.069148 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26f4f5f-7a45-480e-804b-95dc01200ff9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:24.069997 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26f4f5f-7a45-480e-804b-95dc01200ff9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:47:24.070375 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26f4f5f-7a45-480e-804b-95dc01200ff9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:24.072603 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26f4f5f-7a45-480e-804b-95dc01200ff9 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:24.079402 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:24.092055 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1061/2118] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:47:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:47:24.093609 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9bcc70-4f99-4979-b36c-f8e54894f488 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:24.108016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9bcc70-4f99-4979-b36c-f8e54894f488 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:24.114269 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149e7441-3d51-4f2e-b4f0-dac04c886076 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:24.116602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149e7441-3d51-4f2e-b4f0-dac04c886076 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:24.116602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149e7441-3d51-4f2e-b4f0-dac04c886076 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:24.116602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149e7441-3d51-4f2e-b4f0-dac04c886076 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:24.116602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149e7441-3d51-4f2e-b4f0-dac04c886076 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:24.119595 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149e7441-3d51-4f2e-b4f0-dac04c886076 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:24.257977 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1058/2119] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:24 2022] POST /identity/v3/projects => generated 389 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:47:24.258123 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26f4f5f-7a45-480e-804b-95dc01200ff9 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:24.259998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-e26f4f5f-7a45-480e-804b-95dc01200ff9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:24.262423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149e7441-3d51-4f2e-b4f0-dac04c886076 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:24.281190 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26f4f5f-7a45-480e-804b-95dc01200ff9 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:24.322417 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:24.372353 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149e7441-3d51-4f2e-b4f0-dac04c886076 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:24.383916 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565de2ab-4c0a-4d27-b214-954e5dd92189 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:24.384846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565de2ab-4c0a-4d27-b214-954e5dd92189 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:24.385306 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565de2ab-4c0a-4d27-b214-954e5dd92189 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:24.385757 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565de2ab-4c0a-4d27-b214-954e5dd92189 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:24.386652 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565de2ab-4c0a-4d27-b214-954e5dd92189 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:24.389631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565de2ab-4c0a-4d27-b214-954e5dd92189 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:24.398688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a0678a-38ae-4c0d-919e-48b51e2d4e68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:24.399730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a0678a-38ae-4c0d-919e-48b51e2d4e68 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:24.399730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a0678a-38ae-4c0d-919e-48b51e2d4e68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:24.400045 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a0678a-38ae-4c0d-919e-48b51e2d4e68 admin admin] PATH_INFO: `/v3/projects/fcd06564ba0843d6b521120bebe94b95` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:24.401372 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a0678a-38ae-4c0d-919e-48b51e2d4e68 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fcd06564ba0843d6b521120bebe94b95)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:24.403315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a0678a-38ae-4c0d-919e-48b51e2d4e68 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:24.506348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1059/2120] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:24 2022] POST /identity/v3/users => generated 295 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:24.507529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565de2ab-4c0a-4d27-b214-954e5dd92189 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:24.517262 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565de2ab-4c0a-4d27-b214-954e5dd92189 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:24.521876 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad44020b-9e69-436e-a4a3-7efd56889293 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:24.522528 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad44020b-9e69-436e-a4a3-7efd56889293 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:24.522836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad44020b-9e69-436e-a4a3-7efd56889293 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:24.523106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad44020b-9e69-436e-a4a3-7efd56889293 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:24.523739 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad44020b-9e69-436e-a4a3-7efd56889293 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:24.525316 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad44020b-9e69-436e-a4a3-7efd56889293 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:24.536532 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1060/2121] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:24 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:24.544395 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad44020b-9e69-436e-a4a3-7efd56889293 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:24.556020 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad44020b-9e69-436e-a4a3-7efd56889293 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:24.560538 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-58a0678a-38ae-4c0d-919e-48b51e2d4e68 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fcd06564ba0843d6b521120bebe94b95', 'request_id': 'req-58a0678a-38ae-4c0d-919e-48b51e2d4e68'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:24.567369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-58a0678a-38ae-4c0d-919e-48b51e2d4e68 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fcd06564ba0843d6b521120bebe94b95', 'request_id': 'req-58a0678a-38ae-4c0d-919e-48b51e2d4e68'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:24.575352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-58a0678a-38ae-4c0d-919e-48b51e2d4e68 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fcd06564ba0843d6b521120bebe94b95', 'request_id': 'req-58a0678a-38ae-4c0d-919e-48b51e2d4e68'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:24.617503 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c45e78d-4de6-4d35-83d5-622b841900a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:24.618119 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c45e78d-4de6-4d35-83d5-622b841900a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:24.618398 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c45e78d-4de6-4d35-83d5-622b841900a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:24.618699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c45e78d-4de6-4d35-83d5-622b841900a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:24.619596 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c45e78d-4de6-4d35-83d5-622b841900a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:24.621180 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c45e78d-4de6-4d35-83d5-622b841900a4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:24.631381 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1061/2122] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:24 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:47:24.632617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fab811e-4fbc-4983-b768-86ba3471af2c None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:24.632906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fab811e-4fbc-4983-b768-86ba3471af2c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:24.633211 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fab811e-4fbc-4983-b768-86ba3471af2c None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:24.634438 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1062/2123] 10.222.0.39 () {54 vars in 921 bytes} [Fri Sep 30 09:47:24 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:47:24.635119 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fab811e-4fbc-4983-b768-86ba3471af2c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:24.647440 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fab811e-4fbc-4983-b768-86ba3471af2c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:24.653248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525d64e5-7268-4e5b-b862-18784f608353 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:24.656183 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525d64e5-7268-4e5b-b862-18784f608353 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:24.656183 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525d64e5-7268-4e5b-b862-18784f608353 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:24.656183 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525d64e5-7268-4e5b-b862-18784f608353 admin admin] PATH_INFO: `/v3/projects/fa7a5810c8f840f699ba8281407f4a81/users/08e1631cd74247409d471e0cd640d253/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:24.669220 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525d64e5-7268-4e5b-b862-18784f608353 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa7a5810c8f840f699ba8281407f4a81, user_id=08e1631cd74247409d471e0cd640d253, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:24.672765 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525d64e5-7268-4e5b-b862-18784f608353 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:24.677590 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-58a0678a-38ae-4c0d-919e-48b51e2d4e68 admin admin] The token cache is being invalidate because project fcd06564ba0843d6b521120bebe94b95 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:24.683820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-58a0678a-38ae-4c0d-919e-48b51e2d4e68 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:24.683820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-58a0678a-38ae-4c0d-919e-48b51e2d4e68 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:24.683820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1062/2124] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:24 2022] DELETE /identity/v3/projects/fcd06564ba0843d6b521120bebe94b95 => generated 0 bytes in 423 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:24.683820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be26ee65-1caa-4601-96e3-0304e711c694 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:24.683820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be26ee65-1caa-4601-96e3-0304e711c694 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:24.683820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be26ee65-1caa-4601-96e3-0304e711c694 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:24.701163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-be26ee65-1caa-4601-96e3-0304e711c694 None None] MFA Rules not processed for user `49a3011c103b4e48904f876f81d27fba`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:24.702363 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-be26ee65-1caa-4601-96e3-0304e711c694 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:24.768648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1063/2125] 10.222.0.39 () {58 vars in 1018 bytes} [Fri Sep 30 09:47:24 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:24.769589 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be26ee65-1caa-4601-96e3-0304e711c694 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:24.771432 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-be26ee65-1caa-4601-96e3-0304e711c694 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:24.778031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1063/2126] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:24 2022] PUT /identity/v3/projects/fa7a5810c8f840f699ba8281407f4a81/users/08e1631cd74247409d471e0cd640d253/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:24.778031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525d64e5-7268-4e5b-b862-18784f608353 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:24.789055 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-525d64e5-7268-4e5b-b862-18784f608353 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:24.789167 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be26ee65-1caa-4601-96e3-0304e711c694 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:24.849618 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99649ddf-5798-49da-b9b2-10d27c8d0e46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:24.851550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99649ddf-5798-49da-b9b2-10d27c8d0e46 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:24.851838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99649ddf-5798-49da-b9b2-10d27c8d0e46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:24.852118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99649ddf-5798-49da-b9b2-10d27c8d0e46 admin admin] PATH_INFO: `/v3/users/f06ff49709e04623b42db4b0c7612163` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:24.853282 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99649ddf-5798-49da-b9b2-10d27c8d0e46 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f06ff49709e04623b42db4b0c7612163)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:24.855268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99649ddf-5798-49da-b9b2-10d27c8d0e46 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:24.859119 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525d64e5-7268-4e5b-b862-18784f608353 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:24.864006 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4af972-003e-4546-ae41-c76152eb5954 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:24.864702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4af972-003e-4546-ae41-c76152eb5954 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:24.865062 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4af972-003e-4546-ae41-c76152eb5954 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:24.865407 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4af972-003e-4546-ae41-c76152eb5954 admin admin] PATH_INFO: `/v3/users/ae7915e1b2474d9384a0fdcb2ea177b6` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:24.866630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4af972-003e-4546-ae41-c76152eb5954 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ae7915e1b2474d9384a0fdcb2ea177b6)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:24.868544 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4af972-003e-4546-ae41-c76152eb5954 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:25.088084 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-99649ddf-5798-49da-b9b2-10d27c8d0e46 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f06ff49709e04623b42db4b0c7612163'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.099848 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-99649ddf-5798-49da-b9b2-10d27c8d0e46 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f06ff49709e04623b42db4b0c7612163'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.100526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-99649ddf-5798-49da-b9b2-10d27c8d0e46 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f06ff49709e04623b42db4b0c7612163'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.118178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6c4af972-003e-4546-ae41-c76152eb5954 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ae7915e1b2474d9384a0fdcb2ea177b6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.118855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6c4af972-003e-4546-ae41-c76152eb5954 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ae7915e1b2474d9384a0fdcb2ea177b6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.119720 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6c4af972-003e-4546-ae41-c76152eb5954 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ae7915e1b2474d9384a0fdcb2ea177b6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.127469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6c4af972-003e-4546-ae41-c76152eb5954 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ae7915e1b2474d9384a0fdcb2ea177b6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.240104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-99649ddf-5798-49da-b9b2-10d27c8d0e46 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f06ff49709e04623b42db4b0c7612163'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.248256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-99649ddf-5798-49da-b9b2-10d27c8d0e46 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f06ff49709e04623b42db4b0c7612163'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.254228 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1064/2127] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:24 2022] DELETE /identity/v3/users/f06ff49709e04623b42db4b0c7612163 => generated 0 bytes in 485 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:25.255674 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99649ddf-5798-49da-b9b2-10d27c8d0e46 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:25.257377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-99649ddf-5798-49da-b9b2-10d27c8d0e46 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:25.315547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6c4af972-003e-4546-ae41-c76152eb5954 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ae7915e1b2474d9384a0fdcb2ea177b6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.322056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99649ddf-5798-49da-b9b2-10d27c8d0e46 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:25.327439 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e863f1-2fd6-4c23-a42a-4baae1ea00a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:25.327815 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e863f1-2fd6-4c23-a42a-4baae1ea00a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:25.328063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e863f1-2fd6-4c23-a42a-4baae1ea00a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:25.328335 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e863f1-2fd6-4c23-a42a-4baae1ea00a1 admin admin] PATH_INFO: `/v3/projects/59079381c1824e989b2ee73803e4f0f7` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:25.329473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e863f1-2fd6-4c23-a42a-4baae1ea00a1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=59079381c1824e989b2ee73803e4f0f7)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:25.331495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e863f1-2fd6-4c23-a42a-4baae1ea00a1 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:25.345342 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1064/2128] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:24 2022] DELETE /identity/v3/users/ae7915e1b2474d9384a0fdcb2ea177b6 => generated 0 bytes in 569 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:25.346396 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e385f095-df14-43e4-bee8-2792e06aea59 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:25.346700 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e385f095-df14-43e4-bee8-2792e06aea59 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:25.347085 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e385f095-df14-43e4-bee8-2792e06aea59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:25.378413 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e385f095-df14-43e4-bee8-2792e06aea59 None None] MFA Rules not processed for user `08e1631cd74247409d471e0cd640d253`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:25.379793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e385f095-df14-43e4-bee8-2792e06aea59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:25.451882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1065/2129] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:25 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:25.451882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279b7db4-c2b6-456d-86fc-bc61af3e0fdd None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:25.451882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279b7db4-c2b6-456d-86fc-bc61af3e0fdd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:25.451882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279b7db4-c2b6-456d-86fc-bc61af3e0fdd None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:25.451882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1066/2130] 10.222.0.39 () {54 vars in 961 bytes} [Fri Sep 30 09:47:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:47:25.458874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766c8fc2-0f69-4e47-a395-774e58e44ba1 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:25.459221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766c8fc2-0f69-4e47-a395-774e58e44ba1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:25.459461 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766c8fc2-0f69-4e47-a395-774e58e44ba1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:25.471138 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-18e863f1-2fd6-4c23-a42a-4baae1ea00a1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '59079381c1824e989b2ee73803e4f0f7', 'request_id': 'req-18e863f1-2fd6-4c23-a42a-4baae1ea00a1'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.485743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-18e863f1-2fd6-4c23-a42a-4baae1ea00a1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '59079381c1824e989b2ee73803e4f0f7', 'request_id': 'req-18e863f1-2fd6-4c23-a42a-4baae1ea00a1'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.488712 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-766c8fc2-0f69-4e47-a395-774e58e44ba1 None None] MFA Rules not processed for user `49a3011c103b4e48904f876f81d27fba`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:25.489933 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-766c8fc2-0f69-4e47-a395-774e58e44ba1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:25.498631 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-18e863f1-2fd6-4c23-a42a-4baae1ea00a1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '59079381c1824e989b2ee73803e4f0f7', 'request_id': 'req-18e863f1-2fd6-4c23-a42a-4baae1ea00a1'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.564077 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1067/2131] 10.222.0.39 () {58 vars in 1058 bytes} [Fri Sep 30 09:47:25 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:25.565025 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766c8fc2-0f69-4e47-a395-774e58e44ba1 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:25.566682 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-766c8fc2-0f69-4e47-a395-774e58e44ba1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:25.617882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-18e863f1-2fd6-4c23-a42a-4baae1ea00a1 admin admin] The token cache is being invalidate because project 59079381c1824e989b2ee73803e4f0f7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:25.620132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-18e863f1-2fd6-4c23-a42a-4baae1ea00a1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.620132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-18e863f1-2fd6-4c23-a42a-4baae1ea00a1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.621578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1065/2132] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:25 2022] DELETE /identity/v3/projects/59079381c1824e989b2ee73803e4f0f7 => generated 0 bytes in 367 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:25.622532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e863f1-2fd6-4c23-a42a-4baae1ea00a1 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:25.624535 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-18e863f1-2fd6-4c23-a42a-4baae1ea00a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:25.635207 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766c8fc2-0f69-4e47-a395-774e58e44ba1 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:25.643394 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e863f1-2fd6-4c23-a42a-4baae1ea00a1 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:25.643499 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83ff966-78ea-4e8e-b9d0-e22ae7f43a11 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:25.643499 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83ff966-78ea-4e8e-b9d0-e22ae7f43a11 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:25.644409 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83ff966-78ea-4e8e-b9d0-e22ae7f43a11 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:25.645273 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83ff966-78ea-4e8e-b9d0-e22ae7f43a11 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:25.647652 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b83ff966-78ea-4e8e-b9d0-e22ae7f43a11 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:25.647960 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da05902-862d-4a6d-ae47-d512ffb55300 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '49a3011c103b4e48904f876f81d27fba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:25.648915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da05902-862d-4a6d-ae47-d512ffb55300 service placement] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:25.649272 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da05902-862d-4a6d-ae47-d512ffb55300 service placement] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:25.649974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da05902-862d-4a6d-ae47-d512ffb55300 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:25.652105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-8da05902-862d-4a6d-ae47-d512ffb55300 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:25.673043 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da05902-862d-4a6d-ae47-d512ffb55300 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:25.674706 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da05902-862d-4a6d-ae47-d512ffb55300 service placement] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:25.684893 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1066/2133] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:25 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:25.686095 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da05902-862d-4a6d-ae47-d512ffb55300 service placement] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:25.687505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-8da05902-862d-4a6d-ae47-d512ffb55300 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:25.720703 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83ff966-78ea-4e8e-b9d0-e22ae7f43a11 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:25.722400 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83ff966-78ea-4e8e-b9d0-e22ae7f43a11 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:25.732760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1068/2134] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:25 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:25.733981 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efe9f65-e309-4d54-94c3-e920f936f034 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:25.734262 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efe9f65-e309-4d54-94c3-e920f936f034 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:25.734532 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efe9f65-e309-4d54-94c3-e920f936f034 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:25.755823 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-6efe9f65-e309-4d54-94c3-e920f936f034 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:25.757022 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6efe9f65-e309-4d54-94c3-e920f936f034 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:25.759288 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da05902-862d-4a6d-ae47-d512ffb55300 service placement] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:25.765736 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1069/2135] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:25.765864 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede39302-1dfc-4999-a6e1-12766fa59454 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:25.766263 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede39302-1dfc-4999-a6e1-12766fa59454 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:25.766549 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede39302-1dfc-4999-a6e1-12766fa59454 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:25.766857 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede39302-1dfc-4999-a6e1-12766fa59454 admin admin] PATH_INFO: `/v3/projects/b7f6b7ca2f3b4c20ab58940f7e24c1ed` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:25.771428 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede39302-1dfc-4999-a6e1-12766fa59454 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b7f6b7ca2f3b4c20ab58940f7e24c1ed)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:25.773421 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d50e00e-814d-407d-974d-e3b73311e487 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:25.773777 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d50e00e-814d-407d-974d-e3b73311e487 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:25.774123 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d50e00e-814d-407d-974d-e3b73311e487 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:25.774408 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede39302-1dfc-4999-a6e1-12766fa59454 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:25.804987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-0d50e00e-814d-407d-974d-e3b73311e487 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:25.806294 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0d50e00e-814d-407d-974d-e3b73311e487 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:25.815589 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1070/2136] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:47:25.821550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d50e00e-814d-407d-974d-e3b73311e487 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:25.821550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0d50e00e-814d-407d-974d-e3b73311e487 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:25.832600 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d50e00e-814d-407d-974d-e3b73311e487 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:25.837215 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9eac3e-9e0b-4529-a32d-a08a25f65661 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:25.837853 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9eac3e-9e0b-4529-a32d-a08a25f65661 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:25.838130 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9eac3e-9e0b-4529-a32d-a08a25f65661 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:25.838713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9eac3e-9e0b-4529-a32d-a08a25f65661 admin admin] PATH_INFO: `/v3/projects/bb7164960bf24f24b7aed86e9e40130e` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:25.839799 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9eac3e-9e0b-4529-a32d-a08a25f65661 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bb7164960bf24f24b7aed86e9e40130e)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:25.841743 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9eac3e-9e0b-4529-a32d-a08a25f65661 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:25.881260 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:25.885495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-ede39302-1dfc-4999-a6e1-12766fa59454 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b7f6b7ca2f3b4c20ab58940f7e24c1ed', 'request_id': 'req-ede39302-1dfc-4999-a6e1-12766fa59454'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.891423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-ede39302-1dfc-4999-a6e1-12766fa59454 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b7f6b7ca2f3b4c20ab58940f7e24c1ed', 'request_id': 'req-ede39302-1dfc-4999-a6e1-12766fa59454'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.897677 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-ede39302-1dfc-4999-a6e1-12766fa59454 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b7f6b7ca2f3b4c20ab58940f7e24c1ed', 'request_id': 'req-ede39302-1dfc-4999-a6e1-12766fa59454'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.931600 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2f9eac3e-9e0b-4529-a32d-a08a25f65661 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bb7164960bf24f24b7aed86e9e40130e', 'request_id': 'req-2f9eac3e-9e0b-4529-a32d-a08a25f65661'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.937812 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2f9eac3e-9e0b-4529-a32d-a08a25f65661 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bb7164960bf24f24b7aed86e9e40130e', 'request_id': 'req-2f9eac3e-9e0b-4529-a32d-a08a25f65661'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.944769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2f9eac3e-9e0b-4529-a32d-a08a25f65661 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bb7164960bf24f24b7aed86e9e40130e', 'request_id': 'req-2f9eac3e-9e0b-4529-a32d-a08a25f65661'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:25.999315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-ede39302-1dfc-4999-a6e1-12766fa59454 admin admin] The token cache is being invalidate because project b7f6b7ca2f3b4c20ab58940f7e24c1ed was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:26.000642 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-ede39302-1dfc-4999-a6e1-12766fa59454 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:26.000642 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-ede39302-1dfc-4999-a6e1-12766fa59454 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:26.001736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1067/2137] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:25 2022] DELETE /identity/v3/projects/b7f6b7ca2f3b4c20ab58940f7e24c1ed => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:26.002512 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede39302-1dfc-4999-a6e1-12766fa59454 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:26.003872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ede39302-1dfc-4999-a6e1-12766fa59454 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:26.067218 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede39302-1dfc-4999-a6e1-12766fa59454 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:26.079534 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2f9eac3e-9e0b-4529-a32d-a08a25f65661 admin admin] The token cache is being invalidate because project bb7164960bf24f24b7aed86e9e40130e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:26.079899 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2f9eac3e-9e0b-4529-a32d-a08a25f65661 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:26.080556 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2f9eac3e-9e0b-4529-a32d-a08a25f65661 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:26.082281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1071/2138] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:25 2022] DELETE /identity/v3/projects/bb7164960bf24f24b7aed86e9e40130e => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:26.083603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d341db36-a794-4ce5-8bd7-f1ba15d3383f None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:26.083883 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d341db36-a794-4ce5-8bd7-f1ba15d3383f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:26.084159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d341db36-a794-4ce5-8bd7-f1ba15d3383f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:26.105486 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d341db36-a794-4ce5-8bd7-f1ba15d3383f None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:26.106737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d341db36-a794-4ce5-8bd7-f1ba15d3383f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:26.129098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543c40fc-a6a1-4963-b2f5-e7c47fa3b654 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:26.129877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543c40fc-a6a1-4963-b2f5-e7c47fa3b654 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:26.130182 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543c40fc-a6a1-4963-b2f5-e7c47fa3b654 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:26.130442 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543c40fc-a6a1-4963-b2f5-e7c47fa3b654 admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:26.131154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543c40fc-a6a1-4963-b2f5-e7c47fa3b654 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:47:26.131466 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543c40fc-a6a1-4963-b2f5-e7c47fa3b654 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:26.136771 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543c40fc-a6a1-4963-b2f5-e7c47fa3b654 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:26.145843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1068/2139] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:47:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:47:26.147622 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543c40fc-a6a1-4963-b2f5-e7c47fa3b654 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:26.150630 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-543c40fc-a6a1-4963-b2f5-e7c47fa3b654 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:26.164614 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543c40fc-a6a1-4963-b2f5-e7c47fa3b654 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:26.170423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5529bc0b-1353-4e4d-844c-4dbc436f18fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:26.171039 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5529bc0b-1353-4e4d-844c-4dbc436f18fb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:26.171749 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5529bc0b-1353-4e4d-844c-4dbc436f18fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:26.172647 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5529bc0b-1353-4e4d-844c-4dbc436f18fb admin admin] PATH_INFO: `/v3/users/3050bf0a0b014f6db88a28a5963b9de8` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:26.178691 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1072/2140] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:26.178691 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d341db36-a794-4ce5-8bd7-f1ba15d3383f None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:26.179809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5529bc0b-1353-4e4d-844c-4dbc436f18fb admin admin] RBAC: Authorizing `identity:delete_user(user_id=3050bf0a0b014f6db88a28a5963b9de8)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:26.181046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d341db36-a794-4ce5-8bd7-f1ba15d3383f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:26.181840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5529bc0b-1353-4e4d-844c-4dbc436f18fb admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:26.183813 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:26.193537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d341db36-a794-4ce5-8bd7-f1ba15d3383f None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:26.198892 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef65353e-f0b3-425e-b954-9679b3e062b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:26.203844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef65353e-f0b3-425e-b954-9679b3e062b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:26.203844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef65353e-f0b3-425e-b954-9679b3e062b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:26.204006 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef65353e-f0b3-425e-b954-9679b3e062b2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:26.204897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef65353e-f0b3-425e-b954-9679b3e062b2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:26.206867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef65353e-f0b3-425e-b954-9679b3e062b2 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:26.340628 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1073/2141] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:26 2022] POST /identity/v3/projects => generated 383 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:26.342451 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619adb7a-b6f9-4b0f-96ef-705c85bc94b8 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:26.342990 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619adb7a-b6f9-4b0f-96ef-705c85bc94b8 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:26.343270 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619adb7a-b6f9-4b0f-96ef-705c85bc94b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:26.363862 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-619adb7a-b6f9-4b0f-96ef-705c85bc94b8 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:26.373012 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-619adb7a-b6f9-4b0f-96ef-705c85bc94b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:26.424814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1074/2142] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:26.425663 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619adb7a-b6f9-4b0f-96ef-705c85bc94b8 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:26.436408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619adb7a-b6f9-4b0f-96ef-705c85bc94b8 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:26.440706 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5529bc0b-1353-4e4d-844c-4dbc436f18fb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3050bf0a0b014f6db88a28a5963b9de8'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:26.443201 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2233edb7-606f-48c0-bce3-46026ff7a12a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:26.443868 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2233edb7-606f-48c0-bce3-46026ff7a12a admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:26.444148 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2233edb7-606f-48c0-bce3-46026ff7a12a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:26.444433 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2233edb7-606f-48c0-bce3-46026ff7a12a admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:26.445186 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2233edb7-606f-48c0-bce3-46026ff7a12a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:26.447156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2233edb7-606f-48c0-bce3-46026ff7a12a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:26.456246 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5529bc0b-1353-4e4d-844c-4dbc436f18fb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3050bf0a0b014f6db88a28a5963b9de8'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:26.457008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5529bc0b-1353-4e4d-844c-4dbc436f18fb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3050bf0a0b014f6db88a28a5963b9de8'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:26.552726 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1075/2143] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:26 2022] POST /identity/v3/users => generated 292 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:26.553593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2233edb7-606f-48c0-bce3-46026ff7a12a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:26.555230 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2233edb7-606f-48c0-bce3-46026ff7a12a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:26.570458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2233edb7-606f-48c0-bce3-46026ff7a12a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:26.575290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ed51d8-fe4e-485b-adc5-59bee44a1c24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:26.575889 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ed51d8-fe4e-485b-adc5-59bee44a1c24 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:26.576171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ed51d8-fe4e-485b-adc5-59bee44a1c24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:26.576446 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ed51d8-fe4e-485b-adc5-59bee44a1c24 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:26.577092 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ed51d8-fe4e-485b-adc5-59bee44a1c24 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:47:26.577401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ed51d8-fe4e-485b-adc5-59bee44a1c24 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:26.579033 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ed51d8-fe4e-485b-adc5-59bee44a1c24 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:26.582421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5529bc0b-1353-4e4d-844c-4dbc436f18fb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3050bf0a0b014f6db88a28a5963b9de8'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:26.591652 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5529bc0b-1353-4e4d-844c-4dbc436f18fb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3050bf0a0b014f6db88a28a5963b9de8'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:26.592531 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1076/2144] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:47:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:47:26.593513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ed51d8-fe4e-485b-adc5-59bee44a1c24 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:26.595290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a6ed51d8-fe4e-485b-adc5-59bee44a1c24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:26.603112 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1069/2145] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:26 2022] DELETE /identity/v3/users/3050bf0a0b014f6db88a28a5963b9de8 => generated 0 bytes in 457 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:26.603914 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5529bc0b-1353-4e4d-844c-4dbc436f18fb admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:26.612407 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ed51d8-fe4e-485b-adc5-59bee44a1c24 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:26.673554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5529bc0b-1353-4e4d-844c-4dbc436f18fb admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:26.679055 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4905fc9c-fe71-40d8-be61-3955db9a2cab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:26.679675 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4905fc9c-fe71-40d8-be61-3955db9a2cab admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:26.679895 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbc7bc5-0623-4ed9-a559-b121eafe62ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:26.680092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4905fc9c-fe71-40d8-be61-3955db9a2cab admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:26.680301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4905fc9c-fe71-40d8-be61-3955db9a2cab admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:26.680523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbc7bc5-0623-4ed9-a559-b121eafe62ad admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:26.680810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbc7bc5-0623-4ed9-a559-b121eafe62ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:26.681091 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbc7bc5-0623-4ed9-a559-b121eafe62ad admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:26.681216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4905fc9c-fe71-40d8-be61-3955db9a2cab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:26.681740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbc7bc5-0623-4ed9-a559-b121eafe62ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:26.683388 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4905fc9c-fe71-40d8-be61-3955db9a2cab admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:26.683523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbc7bc5-0623-4ed9-a559-b121eafe62ad admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:26.695502 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1077/2146] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:26 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:26.700190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbc7bc5-0623-4ed9-a559-b121eafe62ad admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:26.709853 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbc7bc5-0623-4ed9-a559-b121eafe62ad admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:26.714815 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a754c1b9-3f18-4b1b-b0f2-7ccfbdf56bd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:26.715418 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a754c1b9-3f18-4b1b-b0f2-7ccfbdf56bd7 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:26.715703 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a754c1b9-3f18-4b1b-b0f2-7ccfbdf56bd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:26.715976 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a754c1b9-3f18-4b1b-b0f2-7ccfbdf56bd7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:26.717131 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a754c1b9-3f18-4b1b-b0f2-7ccfbdf56bd7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:26.718935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a754c1b9-3f18-4b1b-b0f2-7ccfbdf56bd7 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:26.732821 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1078/2147] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:26 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:26.734026 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a754c1b9-3f18-4b1b-b0f2-7ccfbdf56bd7 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:26.744586 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a754c1b9-3f18-4b1b-b0f2-7ccfbdf56bd7 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:26.749438 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2276c1d4-799f-4502-9fa7-55fb7cb44110 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:26.750084 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2276c1d4-799f-4502-9fa7-55fb7cb44110 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:26.750363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2276c1d4-799f-4502-9fa7-55fb7cb44110 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:26.750662 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2276c1d4-799f-4502-9fa7-55fb7cb44110 admin admin] PATH_INFO: `/v3/projects/697abea4080645008cdac2b13c2baf71/users/6dab846e5181448fa60585d38c6d71d2/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:26.765547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2276c1d4-799f-4502-9fa7-55fb7cb44110 admin admin] RBAC: Authorizing `identity:create_grant(project_id=697abea4080645008cdac2b13c2baf71, user_id=6dab846e5181448fa60585d38c6d71d2, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:26.768434 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2276c1d4-799f-4502-9fa7-55fb7cb44110 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:26.786630 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1070/2148] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:26 2022] POST /identity/v3/projects => generated 385 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:26.788575 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4905fc9c-fe71-40d8-be61-3955db9a2cab admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:26.854431 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4905fc9c-fe71-40d8-be61-3955db9a2cab admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:26.859747 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326f9bde-754c-46df-8fda-255e3d6406e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:26.860353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326f9bde-754c-46df-8fda-255e3d6406e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:26.860633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326f9bde-754c-46df-8fda-255e3d6406e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:26.860913 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326f9bde-754c-46df-8fda-255e3d6406e4 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:26.861534 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326f9bde-754c-46df-8fda-255e3d6406e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:26.863588 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326f9bde-754c-46df-8fda-255e3d6406e4 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:26.870343 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1079/2149] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:26 2022] PUT /identity/v3/projects/697abea4080645008cdac2b13c2baf71/users/6dab846e5181448fa60585d38c6d71d2/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:47:26.881285 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97d5cc8-9019-46c9-bd53-aab68232c782 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:26.881575 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97d5cc8-9019-46c9-bd53-aab68232c782 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:26.881849 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97d5cc8-9019-46c9-bd53-aab68232c782 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:26.911640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b97d5cc8-9019-46c9-bd53-aab68232c782 None None] MFA Rules not processed for user `6dab846e5181448fa60585d38c6d71d2`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:26.912863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b97d5cc8-9019-46c9-bd53-aab68232c782 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:26.950687 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1071/2150] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:26 2022] POST /identity/v3/users => generated 293 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:47:26.953968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326f9bde-754c-46df-8fda-255e3d6406e4 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:26.965385 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326f9bde-754c-46df-8fda-255e3d6406e4 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:26.985294 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1080/2151] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:26 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:47:26.988100 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97d5cc8-9019-46c9-bd53-aab68232c782 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:27.042737 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1693f98-71c7-4201-8a14-d766bcdef5e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:27.042737 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1693f98-71c7-4201-8a14-d766bcdef5e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:27.042737 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1693f98-71c7-4201-8a14-d766bcdef5e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:27.042737 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1693f98-71c7-4201-8a14-d766bcdef5e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:27.043119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1693f98-71c7-4201-8a14-d766bcdef5e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:27.043119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1693f98-71c7-4201-8a14-d766bcdef5e9 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:27.053690 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1072/2152] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:26 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:47:27.053690 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1693f98-71c7-4201-8a14-d766bcdef5e9 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:27.053690 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-e1693f98-71c7-4201-8a14-d766bcdef5e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:27.060142 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97d5cc8-9019-46c9-bd53-aab68232c782 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:27.065665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef9d21b-1d5e-4999-801c-960e745bafa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:27.066279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef9d21b-1d5e-4999-801c-960e745bafa8 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:27.066559 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef9d21b-1d5e-4999-801c-960e745bafa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:27.066696 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1693f98-71c7-4201-8a14-d766bcdef5e9 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:27.074216 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef9d21b-1d5e-4999-801c-960e745bafa8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:27.074216 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef9d21b-1d5e-4999-801c-960e745bafa8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:27.074216 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef9d21b-1d5e-4999-801c-960e745bafa8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:27.074829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161ec54b-015d-4278-9f64-810c401a18cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:27.074829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161ec54b-015d-4278-9f64-810c401a18cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:27.074829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161ec54b-015d-4278-9f64-810c401a18cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:27.074829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161ec54b-015d-4278-9f64-810c401a18cc admin admin] PATH_INFO: `/v3/projects/578897d982ff44169a17f9d384c801d1` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:27.075355 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161ec54b-015d-4278-9f64-810c401a18cc admin admin] RBAC: Authorizing `identity:delete_project(project_id=578897d982ff44169a17f9d384c801d1)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:27.077133 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161ec54b-015d-4278-9f64-810c401a18cc admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:27.090581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:27.179158 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1081/2153] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:26 2022] POST /identity/v3/projects => generated 383 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:47:27.180494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef9d21b-1d5e-4999-801c-960e745bafa8 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:27.198340 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-161ec54b-015d-4278-9f64-810c401a18cc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '578897d982ff44169a17f9d384c801d1', 'request_id': 'req-161ec54b-015d-4278-9f64-810c401a18cc'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:27.206075 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-161ec54b-015d-4278-9f64-810c401a18cc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '578897d982ff44169a17f9d384c801d1', 'request_id': 'req-161ec54b-015d-4278-9f64-810c401a18cc'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:27.213093 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-161ec54b-015d-4278-9f64-810c401a18cc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '578897d982ff44169a17f9d384c801d1', 'request_id': 'req-161ec54b-015d-4278-9f64-810c401a18cc'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:27.247635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef9d21b-1d5e-4999-801c-960e745bafa8 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:27.303274 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a995593-8341-4327-8c58-18b68e6ee6f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:27.303806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a995593-8341-4327-8c58-18b68e6ee6f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:27.304087 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a995593-8341-4327-8c58-18b68e6ee6f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:27.304362 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a995593-8341-4327-8c58-18b68e6ee6f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:27.305002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a995593-8341-4327-8c58-18b68e6ee6f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:27.306623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a995593-8341-4327-8c58-18b68e6ee6f4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:27.313557 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-161ec54b-015d-4278-9f64-810c401a18cc admin admin] The token cache is being invalidate because project 578897d982ff44169a17f9d384c801d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:27.313922 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-161ec54b-015d-4278-9f64-810c401a18cc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:27.314677 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-161ec54b-015d-4278-9f64-810c401a18cc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:27.316392 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1073/2154] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:27 2022] DELETE /identity/v3/projects/578897d982ff44169a17f9d384c801d1 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:27.317067 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161ec54b-015d-4278-9f64-810c401a18cc admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:27.318440 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-161ec54b-015d-4278-9f64-810c401a18cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:27.324610 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1082/2155] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:27 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:27.324610 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f98023-ef1f-4cad-938a-480ed22e3fcd None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:27.324610 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f98023-ef1f-4cad-938a-480ed22e3fcd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:27.324610 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f98023-ef1f-4cad-938a-480ed22e3fcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:27.342082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161ec54b-015d-4278-9f64-810c401a18cc admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:27.342082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc374d28-306b-4ace-a006-94176a515bfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:27.342082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc374d28-306b-4ace-a006-94176a515bfc admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:27.342082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc374d28-306b-4ace-a006-94176a515bfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:27.342711 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc374d28-306b-4ace-a006-94176a515bfc admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:27.342984 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc374d28-306b-4ace-a006-94176a515bfc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:27.344924 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc374d28-306b-4ace-a006-94176a515bfc admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:27.360548 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:27.363576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-57f98023-ef1f-4cad-938a-480ed22e3fcd None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:27.365716 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-57f98023-ef1f-4cad-938a-480ed22e3fcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:27.376164 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1083/2156] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:27.377137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f98023-ef1f-4cad-938a-480ed22e3fcd None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:27.378723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-57f98023-ef1f-4cad-938a-480ed22e3fcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:27.391894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f98023-ef1f-4cad-938a-480ed22e3fcd None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:27.397013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38eed5f-e0cc-455c-a36c-1e0fa79b4b69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:27.397820 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38eed5f-e0cc-455c-a36c-1e0fa79b4b69 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:27.398359 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38eed5f-e0cc-455c-a36c-1e0fa79b4b69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:27.398899 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38eed5f-e0cc-455c-a36c-1e0fa79b4b69 admin admin] PATH_INFO: `/v3/projects/0860efc0d2584be78c2b46f937ebe7cf/users/6f80ef9b0f0440eeade62a9b6070db35/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:27.416917 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38eed5f-e0cc-455c-a36c-1e0fa79b4b69 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0860efc0d2584be78c2b46f937ebe7cf, user_id=6f80ef9b0f0440eeade62a9b6070db35, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:27.419335 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38eed5f-e0cc-455c-a36c-1e0fa79b4b69 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:27.441532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1074/2157] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:27 2022] POST /identity/v3/users => generated 298 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:27.441532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7077999b-2a91-41f7-aaad-4b2e47a238be None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:27.441532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7077999b-2a91-41f7-aaad-4b2e47a238be None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:27.441532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7077999b-2a91-41f7-aaad-4b2e47a238be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:27.476434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-7077999b-2a91-41f7-aaad-4b2e47a238be None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:27.478499 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7077999b-2a91-41f7-aaad-4b2e47a238be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:27.490632 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1075/2158] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:27.490632 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7077999b-2a91-41f7-aaad-4b2e47a238be None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:27.501141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7077999b-2a91-41f7-aaad-4b2e47a238be None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:27.519760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1084/2159] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:27 2022] PUT /identity/v3/projects/0860efc0d2584be78c2b46f937ebe7cf/users/6f80ef9b0f0440eeade62a9b6070db35/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:27.519760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38eed5f-e0cc-455c-a36c-1e0fa79b4b69 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:27.607447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73c986d-fa8b-421a-99c7-b50dc00275d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:27.608510 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73c986d-fa8b-421a-99c7-b50dc00275d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:27.608510 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73c986d-fa8b-421a-99c7-b50dc00275d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:27.608777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73c986d-fa8b-421a-99c7-b50dc00275d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:27.609434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73c986d-fa8b-421a-99c7-b50dc00275d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:27.611491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73c986d-fa8b-421a-99c7-b50dc00275d8 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:27.624012 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1076/2160] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:27 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:27.625334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de22d5b5-b6ac-4a2e-a971-2b947ec1edfc None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:27.625638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de22d5b5-b6ac-4a2e-a971-2b947ec1edfc None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:27.625961 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de22d5b5-b6ac-4a2e-a971-2b947ec1edfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:27.629091 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38eed5f-e0cc-455c-a36c-1e0fa79b4b69 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:27.633821 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341a20ec-631e-4333-8271-3073fad5a82c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:27.634433 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341a20ec-631e-4333-8271-3073fad5a82c admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:27.634747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341a20ec-631e-4333-8271-3073fad5a82c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:27.635022 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341a20ec-631e-4333-8271-3073fad5a82c admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:27.635707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341a20ec-631e-4333-8271-3073fad5a82c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:47:27.636048 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341a20ec-631e-4333-8271-3073fad5a82c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:27.637644 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341a20ec-631e-4333-8271-3073fad5a82c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:27.649879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1085/2161] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:47:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:47:27.650703 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341a20ec-631e-4333-8271-3073fad5a82c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:27.654850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-de22d5b5-b6ac-4a2e-a971-2b947ec1edfc None None] MFA Rules not processed for user `6f80ef9b0f0440eeade62a9b6070db35`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:27.656053 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-de22d5b5-b6ac-4a2e-a971-2b947ec1edfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:27.660400 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341a20ec-631e-4333-8271-3073fad5a82c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:27.665493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5044da26-deff-475e-8bcd-a4487a9df7f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:27.666169 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5044da26-deff-475e-8bcd-a4487a9df7f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:27.666451 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5044da26-deff-475e-8bcd-a4487a9df7f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:27.666760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5044da26-deff-475e-8bcd-a4487a9df7f4 admin admin] PATH_INFO: `/v3/projects/68a56179fff34e0bac65384b683e7b83/users/5dc0c4d258fd4ac498ecea94ae1ecdd8/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:27.688899 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5044da26-deff-475e-8bcd-a4487a9df7f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=68a56179fff34e0bac65384b683e7b83, user_id=5dc0c4d258fd4ac498ecea94ae1ecdd8, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:27.691678 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5044da26-deff-475e-8bcd-a4487a9df7f4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:27.721992 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1077/2162] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:27 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:27.722851 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de22d5b5-b6ac-4a2e-a971-2b947ec1edfc None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:27.734478 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de22d5b5-b6ac-4a2e-a971-2b947ec1edfc None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:27.739157 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18702a55-8a2f-4f0f-a18c-f167832aa34c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:27.739741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18702a55-8a2f-4f0f-a18c-f167832aa34c admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:27.740021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18702a55-8a2f-4f0f-a18c-f167832aa34c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:27.740294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18702a55-8a2f-4f0f-a18c-f167832aa34c admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:27.740919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18702a55-8a2f-4f0f-a18c-f167832aa34c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:27.742869 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18702a55-8a2f-4f0f-a18c-f167832aa34c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:27.788913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1086/2163] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:27 2022] PUT /identity/v3/projects/68a56179fff34e0bac65384b683e7b83/users/5dc0c4d258fd4ac498ecea94ae1ecdd8/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:27.789737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5044da26-deff-475e-8bcd-a4487a9df7f4 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:27.791325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5044da26-deff-475e-8bcd-a4487a9df7f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:27.868364 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5044da26-deff-475e-8bcd-a4487a9df7f4 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:27.875177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b6aae0-7cda-414c-9360-59cf8f9b7266 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:27.875784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b6aae0-7cda-414c-9360-59cf8f9b7266 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:27.876068 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b6aae0-7cda-414c-9360-59cf8f9b7266 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:27.876348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b6aae0-7cda-414c-9360-59cf8f9b7266 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:27.885608 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1078/2164] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:27 2022] POST /identity/v3/projects => generated 383 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:27.885608 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18702a55-8a2f-4f0f-a18c-f167832aa34c admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:27.952947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b6aae0-7cda-414c-9360-59cf8f9b7266 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:27.954678 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b6aae0-7cda-414c-9360-59cf8f9b7266 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:27.958967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18702a55-8a2f-4f0f-a18c-f167832aa34c admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:27.963923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfdc288-86f3-412b-b0d3-2b564d308e92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:27.964088 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1087/2165] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:27 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:27.964714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfdc288-86f3-412b-b0d3-2b564d308e92 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:27.964864 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b6aae0-7cda-414c-9360-59cf8f9b7266 service nova] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:27.965020 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfdc288-86f3-412b-b0d3-2b564d308e92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:27.965462 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfdc288-86f3-412b-b0d3-2b564d308e92 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:27.966906 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfdc288-86f3-412b-b0d3-2b564d308e92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:27.968675 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfdc288-86f3-412b-b0d3-2b564d308e92 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:27.979410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1079/2166] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:27 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:27.985346 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfdc288-86f3-412b-b0d3-2b564d308e92 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:27.995189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfdc288-86f3-412b-b0d3-2b564d308e92 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:28.000407 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178b0d71-3a26-4e1f-a07f-0c19c54dc2af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:28.002922 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178b0d71-3a26-4e1f-a07f-0c19c54dc2af admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:28.002922 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178b0d71-3a26-4e1f-a07f-0c19c54dc2af admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:28.002922 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178b0d71-3a26-4e1f-a07f-0c19c54dc2af admin admin] PATH_INFO: `/v3/projects/68a56179fff34e0bac65384b683e7b83/users/5dc0c4d258fd4ac498ecea94ae1ecdd8/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:28.003453 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178b0d71-3a26-4e1f-a07f-0c19c54dc2af admin admin] RBAC: Authorizing `identity:create_grant(project_id=68a56179fff34e0bac65384b683e7b83, user_id=5dc0c4d258fd4ac498ecea94ae1ecdd8, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:28.006098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178b0d71-3a26-4e1f-a07f-0c19c54dc2af admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:28.016720 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1080/2167] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:27 2022] PUT /identity/v3/projects/68a56179fff34e0bac65384b683e7b83/users/5dc0c4d258fd4ac498ecea94ae1ecdd8/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:47:28.026741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd798d0f-ee32-4b5e-80b0-4848ebb1f4fa None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:28.027105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd798d0f-ee32-4b5e-80b0-4848ebb1f4fa None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:28.027486 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd798d0f-ee32-4b5e-80b0-4848ebb1f4fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:28.029879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b6aae0-7cda-414c-9360-59cf8f9b7266 service nova] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:28.061644 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-cd798d0f-ee32-4b5e-80b0-4848ebb1f4fa None None] MFA Rules not processed for user `5dc0c4d258fd4ac498ecea94ae1ecdd8`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:28.061644 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-cd798d0f-ee32-4b5e-80b0-4848ebb1f4fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:28.149280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1081/2168] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:28 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:47:28.237867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e4b43b-04f6-4e35-b1bb-6c572a8da9dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:28.238540 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e4b43b-04f6-4e35-b1bb-6c572a8da9dc admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:28.238891 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e4b43b-04f6-4e35-b1bb-6c572a8da9dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:28.239167 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e4b43b-04f6-4e35-b1bb-6c572a8da9dc admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:28.239885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e4b43b-04f6-4e35-b1bb-6c572a8da9dc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:28.242026 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e4b43b-04f6-4e35-b1bb-6c572a8da9dc admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:28.252068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd798d0f-ee32-4b5e-80b0-4848ebb1f4fa None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:28.253515 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-cd798d0f-ee32-4b5e-80b0-4848ebb1f4fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:28.311322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd798d0f-ee32-4b5e-80b0-4848ebb1f4fa None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:28.315127 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b50b84b-de35-4a9e-9346-a28a48e50964 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:28.315731 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b50b84b-de35-4a9e-9346-a28a48e50964 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:28.316011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b50b84b-de35-4a9e-9346-a28a48e50964 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:28.316282 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b50b84b-de35-4a9e-9346-a28a48e50964 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:28.327943 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b50b84b-de35-4a9e-9346-a28a48e50964 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:28.329488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b50b84b-de35-4a9e-9346-a28a48e50964 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:28.332271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1088/2169] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:27 2022] POST /identity/v3/users => generated 292 bytes in 368 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:28.337550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e4b43b-04f6-4e35-b1bb-6c572a8da9dc admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:28.340409 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1082/2170] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:28 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:47:28.347476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e4b43b-04f6-4e35-b1bb-6c572a8da9dc admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:28.352451 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93dc67cd-9bcf-43e7-9a95-94bcd4e20d11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:28.353110 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dc67cd-9bcf-43e7-9a95-94bcd4e20d11 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:28.353394 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dc67cd-9bcf-43e7-9a95-94bcd4e20d11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:28.353696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dc67cd-9bcf-43e7-9a95-94bcd4e20d11 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:28.354371 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93dc67cd-9bcf-43e7-9a95-94bcd4e20d11 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:28.356004 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93dc67cd-9bcf-43e7-9a95-94bcd4e20d11 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:28.370754 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1089/2171] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:47:28.371361 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b50b84b-de35-4a9e-9346-a28a48e50964 service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:28.382740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b50b84b-de35-4a9e-9346-a28a48e50964 service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:28.387840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606e2486-3753-4530-91da-94b3a68014b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:28.389049 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606e2486-3753-4530-91da-94b3a68014b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:28.389309 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606e2486-3753-4530-91da-94b3a68014b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:28.389566 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606e2486-3753-4530-91da-94b3a68014b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:28.390197 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606e2486-3753-4530-91da-94b3a68014b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:28.393978 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606e2486-3753-4530-91da-94b3a68014b8 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:28.402126 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1083/2172] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:28.406558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93dc67cd-9bcf-43e7-9a95-94bcd4e20d11 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:28.416742 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93dc67cd-9bcf-43e7-9a95-94bcd4e20d11 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:28.421467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e37403-f702-49a5-b152-945b6dec23b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:28.422134 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e37403-f702-49a5-b152-945b6dec23b5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:28.422400 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e37403-f702-49a5-b152-945b6dec23b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:28.422723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e37403-f702-49a5-b152-945b6dec23b5 admin admin] PATH_INFO: `/v3/projects/8e87f05b71744abb926f7f4aad471e8d/users/91282b2827cd4d6e8b5bacb98cd4923a/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:28.444467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e37403-f702-49a5-b152-945b6dec23b5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e87f05b71744abb926f7f4aad471e8d, user_id=91282b2827cd4d6e8b5bacb98cd4923a, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:28.447139 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e37403-f702-49a5-b152-945b6dec23b5 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:28.569627 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1090/2173] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:28 2022] PUT /identity/v3/projects/8e87f05b71744abb926f7f4aad471e8d/users/91282b2827cd4d6e8b5bacb98cd4923a/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:28.582501 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13586c0a-bb1a-4d62-b9ba-c0f141f9196c None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:28.583000 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13586c0a-bb1a-4d62-b9ba-c0f141f9196c None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:28.583266 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13586c0a-bb1a-4d62-b9ba-c0f141f9196c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:28.615306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-13586c0a-bb1a-4d62-b9ba-c0f141f9196c None None] MFA Rules not processed for user `91282b2827cd4d6e8b5bacb98cd4923a`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:28.616541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-13586c0a-bb1a-4d62-b9ba-c0f141f9196c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:28.682799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1084/2174] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:28 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:28.760015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:28.762658 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e37403-f702-49a5-b152-945b6dec23b5 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:28.816668 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e37403-f702-49a5-b152-945b6dec23b5 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:28.820284 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becc0f70-9dc4-4dfb-bdc8-5faf2e1cc73d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:28.820877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becc0f70-9dc4-4dfb-bdc8-5faf2e1cc73d service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:28.821151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becc0f70-9dc4-4dfb-bdc8-5faf2e1cc73d service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:28.821419 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becc0f70-9dc4-4dfb-bdc8-5faf2e1cc73d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:28.823269 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-becc0f70-9dc4-4dfb-bdc8-5faf2e1cc73d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:28.866741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13586c0a-bb1a-4d62-b9ba-c0f141f9196c None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:28.868353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-13586c0a-bb1a-4d62-b9ba-c0f141f9196c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:28.880683 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13586c0a-bb1a-4d62-b9ba-c0f141f9196c None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:28.884588 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06fb3c3-b3ef-44ec-bf5d-b7e912bc5a3b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:28.885315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06fb3c3-b3ef-44ec-bf5d-b7e912bc5a3b service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:28.885596 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06fb3c3-b3ef-44ec-bf5d-b7e912bc5a3b service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:28.885868 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06fb3c3-b3ef-44ec-bf5d-b7e912bc5a3b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:28.886957 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-becc0f70-9dc4-4dfb-bdc8-5faf2e1cc73d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:28.888457 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-becc0f70-9dc4-4dfb-bdc8-5faf2e1cc73d service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:28.894389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e06fb3c3-b3ef-44ec-bf5d-b7e912bc5a3b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:28.895848 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e06fb3c3-b3ef-44ec-bf5d-b7e912bc5a3b service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:28.897890 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1091/2175] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:28 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:28.905401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1085/2176] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:28 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:47:28.926314 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becc0f70-9dc4-4dfb-bdc8-5faf2e1cc73d service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:28.954126 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:29.018120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becc0f70-9dc4-4dfb-bdc8-5faf2e1cc73d service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:29.023351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947667ec-dc8f-4c0e-adf8-48dd2b0179ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:29.024062 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947667ec-dc8f-4c0e-adf8-48dd2b0179ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:29.024407 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947667ec-dc8f-4c0e-adf8-48dd2b0179ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:29.024690 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947667ec-dc8f-4c0e-adf8-48dd2b0179ce admin admin] PATH_INFO: `/v3/users/91282b2827cd4d6e8b5bacb98cd4923a` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:29.026155 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947667ec-dc8f-4c0e-adf8-48dd2b0179ce admin admin] RBAC: Authorizing `identity:delete_user(user_id=91282b2827cd4d6e8b5bacb98cd4923a)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:29.028187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947667ec-dc8f-4c0e-adf8-48dd2b0179ce admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:29.270305 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-947667ec-dc8f-4c0e-adf8-48dd2b0179ce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '91282b2827cd4d6e8b5bacb98cd4923a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:29.270989 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-947667ec-dc8f-4c0e-adf8-48dd2b0179ce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '91282b2827cd4d6e8b5bacb98cd4923a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:29.271661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-947667ec-dc8f-4c0e-adf8-48dd2b0179ce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '91282b2827cd4d6e8b5bacb98cd4923a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:29.279731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-947667ec-dc8f-4c0e-adf8-48dd2b0179ce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '91282b2827cd4d6e8b5bacb98cd4923a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:29.357378 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-947667ec-dc8f-4c0e-adf8-48dd2b0179ce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '91282b2827cd4d6e8b5bacb98cd4923a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:29.375128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1092/2177] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:28 2022] DELETE /identity/v3/users/91282b2827cd4d6e8b5bacb98cd4923a => generated 0 bytes in 449 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:29.379645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192c6994-a8b0-4b7c-a73d-7b66c97e6bda None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:29.379915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192c6994-a8b0-4b7c-a73d-7b66c97e6bda None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:29.380189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192c6994-a8b0-4b7c-a73d-7b66c97e6bda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:29.399157 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-192c6994-a8b0-4b7c-a73d-7b66c97e6bda None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:29.400345 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-192c6994-a8b0-4b7c-a73d-7b66c97e6bda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:29.457935 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1086/2178] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:29.468449 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947667ec-dc8f-4c0e-adf8-48dd2b0179ce admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:29.469874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-947667ec-dc8f-4c0e-adf8-48dd2b0179ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:29.476051 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192c6994-a8b0-4b7c-a73d-7b66c97e6bda None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:29.477439 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-192c6994-a8b0-4b7c-a73d-7b66c97e6bda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:29.489958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192c6994-a8b0-4b7c-a73d-7b66c97e6bda None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:29.494995 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8dbdf5-dd57-45cc-a00e-06c8511bbc44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:29.495620 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8dbdf5-dd57-45cc-a00e-06c8511bbc44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:29.495893 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8dbdf5-dd57-45cc-a00e-06c8511bbc44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:29.496162 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8dbdf5-dd57-45cc-a00e-06c8511bbc44 admin admin] PATH_INFO: `/v3/users/6dab846e5181448fa60585d38c6d71d2` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:29.497316 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8dbdf5-dd57-45cc-a00e-06c8511bbc44 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6dab846e5181448fa60585d38c6d71d2)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:29.499405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8dbdf5-dd57-45cc-a00e-06c8511bbc44 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:29.531529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947667ec-dc8f-4c0e-adf8-48dd2b0179ce admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:29.535272 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10287b3-e491-4b40-b2de-60e1315fb0f2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:29.536020 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10287b3-e491-4b40-b2de-60e1315fb0f2 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:29.536305 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10287b3-e491-4b40-b2de-60e1315fb0f2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:29.536580 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10287b3-e491-4b40-b2de-60e1315fb0f2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:29.545668 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10287b3-e491-4b40-b2de-60e1315fb0f2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:29.547124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10287b3-e491-4b40-b2de-60e1315fb0f2 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:29.556790 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1093/2179] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:29.699402 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7c8dbdf5-dd57-45cc-a00e-06c8511bbc44 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6dab846e5181448fa60585d38c6d71d2'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:29.711996 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7c8dbdf5-dd57-45cc-a00e-06c8511bbc44 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6dab846e5181448fa60585d38c6d71d2'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:29.712812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7c8dbdf5-dd57-45cc-a00e-06c8511bbc44 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6dab846e5181448fa60585d38c6d71d2'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:29.798101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7c8dbdf5-dd57-45cc-a00e-06c8511bbc44 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6dab846e5181448fa60585d38c6d71d2'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:29.807905 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:29.810859 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7c8dbdf5-dd57-45cc-a00e-06c8511bbc44 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6dab846e5181448fa60585d38c6d71d2'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:29.817643 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1087/2180] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:29 2022] DELETE /identity/v3/users/6dab846e5181448fa60585d38c6d71d2 => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:47:29.822736 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984e0e50-4bf2-494f-b2bd-11a27808d40a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:29.823091 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984e0e50-4bf2-494f-b2bd-11a27808d40a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:29.823635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984e0e50-4bf2-494f-b2bd-11a27808d40a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:29.843840 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-984e0e50-4bf2-494f-b2bd-11a27808d40a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:29.845065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-984e0e50-4bf2-494f-b2bd-11a27808d40a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:29.864532 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:29.908699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1094/2181] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:47:29.919162 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8dbdf5-dd57-45cc-a00e-06c8511bbc44 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:29.920875 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7c8dbdf5-dd57-45cc-a00e-06c8511bbc44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:29.985490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8dbdf5-dd57-45cc-a00e-06c8511bbc44 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:29.989165 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21441bb-5899-46dc-b7c1-5d948d80a4c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:29.989768 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21441bb-5899-46dc-b7c1-5d948d80a4c7 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:29.997865 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21441bb-5899-46dc-b7c1-5d948d80a4c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:29.997865 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21441bb-5899-46dc-b7c1-5d948d80a4c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:30.000050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21441bb-5899-46dc-b7c1-5d948d80a4c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:30.001483 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21441bb-5899-46dc-b7c1-5d948d80a4c7 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:30.010791 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1088/2182] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:30.053209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984e0e50-4bf2-494f-b2bd-11a27808d40a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:30.054741 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-984e0e50-4bf2-494f-b2bd-11a27808d40a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:30.068752 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984e0e50-4bf2-494f-b2bd-11a27808d40a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:30.073517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f527a2e5-a65b-4565-b6b7-ed2b7e3af8d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:30.074231 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f527a2e5-a65b-4565-b6b7-ed2b7e3af8d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:30.074613 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f527a2e5-a65b-4565-b6b7-ed2b7e3af8d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:30.074974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f527a2e5-a65b-4565-b6b7-ed2b7e3af8d9 admin admin] PATH_INFO: `/v3/projects/8e87f05b71744abb926f7f4aad471e8d` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:30.076782 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f527a2e5-a65b-4565-b6b7-ed2b7e3af8d9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e87f05b71744abb926f7f4aad471e8d)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:30.078843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f527a2e5-a65b-4565-b6b7-ed2b7e3af8d9 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:30.189015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:30.189359 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f527a2e5-a65b-4565-b6b7-ed2b7e3af8d9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e87f05b71744abb926f7f4aad471e8d', 'request_id': 'req-f527a2e5-a65b-4565-b6b7-ed2b7e3af8d9'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:30.194916 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f527a2e5-a65b-4565-b6b7-ed2b7e3af8d9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e87f05b71744abb926f7f4aad471e8d', 'request_id': 'req-f527a2e5-a65b-4565-b6b7-ed2b7e3af8d9'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:30.201464 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f527a2e5-a65b-4565-b6b7-ed2b7e3af8d9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e87f05b71744abb926f7f4aad471e8d', 'request_id': 'req-f527a2e5-a65b-4565-b6b7-ed2b7e3af8d9'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:30.297275 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:30.333748 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f527a2e5-a65b-4565-b6b7-ed2b7e3af8d9 admin admin] The token cache is being invalidate because project 8e87f05b71744abb926f7f4aad471e8d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:30.334567 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f527a2e5-a65b-4565-b6b7-ed2b7e3af8d9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:30.334973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f527a2e5-a65b-4565-b6b7-ed2b7e3af8d9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:30.336924 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1095/2183] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:30 2022] DELETE /identity/v3/projects/8e87f05b71744abb926f7f4aad471e8d => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:30.341588 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c3ee76-d73e-43ef-87ad-16b8194d32dc None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:30.341875 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c3ee76-d73e-43ef-87ad-16b8194d32dc None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:30.342150 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c3ee76-d73e-43ef-87ad-16b8194d32dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:30.372156 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-68c3ee76-d73e-43ef-87ad-16b8194d32dc None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:30.374121 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-68c3ee76-d73e-43ef-87ad-16b8194d32dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:30.433141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1089/2184] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:47:30.441525 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcf6d91-6f28-4814-915e-a2b700656939 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:30.441808 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcf6d91-6f28-4814-915e-a2b700656939 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:30.442080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcf6d91-6f28-4814-915e-a2b700656939 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:30.462209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-bfcf6d91-6f28-4814-915e-a2b700656939 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:30.463614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-bfcf6d91-6f28-4814-915e-a2b700656939 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:30.471878 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1096/2185] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:30.476321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c3ee76-d73e-43ef-87ad-16b8194d32dc None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:30.485662 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcf6d91-6f28-4814-915e-a2b700656939 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:30.487056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-bfcf6d91-6f28-4814-915e-a2b700656939 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:30.487495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c3ee76-d73e-43ef-87ad-16b8194d32dc None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:30.492638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058b0253-f863-480a-8b6d-f6c6c5b5d740 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:30.493492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058b0253-f863-480a-8b6d-f6c6c5b5d740 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:30.493867 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058b0253-f863-480a-8b6d-f6c6c5b5d740 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:30.494504 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058b0253-f863-480a-8b6d-f6c6c5b5d740 admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:30.495283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-058b0253-f863-480a-8b6d-f6c6c5b5d740 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:47:30.495920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-058b0253-f863-480a-8b6d-f6c6c5b5d740 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:30.497666 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-058b0253-f863-480a-8b6d-f6c6c5b5d740 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:30.502328 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcf6d91-6f28-4814-915e-a2b700656939 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:30.507644 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa55eb7-4edb-4dc3-bb32-85d46f857808 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:30.510139 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa55eb7-4edb-4dc3-bb32-85d46f857808 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:30.510139 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa55eb7-4edb-4dc3-bb32-85d46f857808 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:30.510139 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa55eb7-4edb-4dc3-bb32-85d46f857808 admin admin] PATH_INFO: `/v3/projects/697abea4080645008cdac2b13c2baf71` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:30.511095 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1090/2186] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:47:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:47:30.514936 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa55eb7-4edb-4dc3-bb32-85d46f857808 admin admin] RBAC: Authorizing `identity:delete_project(project_id=697abea4080645008cdac2b13c2baf71)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:30.515103 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058b0253-f863-480a-8b6d-f6c6c5b5d740 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:30.517121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa55eb7-4edb-4dc3-bb32-85d46f857808 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:30.524677 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058b0253-f863-480a-8b6d-f6c6c5b5d740 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:30.530939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14b6179-b419-479c-b9e2-2040393152eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:30.531609 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14b6179-b419-479c-b9e2-2040393152eb admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:30.531947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14b6179-b419-479c-b9e2-2040393152eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:30.532249 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14b6179-b419-479c-b9e2-2040393152eb admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:30.532877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e14b6179-b419-479c-b9e2-2040393152eb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:30.534833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e14b6179-b419-479c-b9e2-2040393152eb admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:30.667015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1091/2187] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:30 2022] POST /identity/v3/projects => generated 385 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:30.671240 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14b6179-b419-479c-b9e2-2040393152eb admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:30.677749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0aa55eb7-4edb-4dc3-bb32-85d46f857808 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '697abea4080645008cdac2b13c2baf71', 'request_id': 'req-0aa55eb7-4edb-4dc3-bb32-85d46f857808'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:30.686854 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0aa55eb7-4edb-4dc3-bb32-85d46f857808 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '697abea4080645008cdac2b13c2baf71', 'request_id': 'req-0aa55eb7-4edb-4dc3-bb32-85d46f857808'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:30.690357 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0aa55eb7-4edb-4dc3-bb32-85d46f857808 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '697abea4080645008cdac2b13c2baf71', 'request_id': 'req-0aa55eb7-4edb-4dc3-bb32-85d46f857808'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:30.734420 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14b6179-b419-479c-b9e2-2040393152eb admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:30.739321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2d7d52-42c6-4b60-86db-7f544cbe0d9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:30.739923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2d7d52-42c6-4b60-86db-7f544cbe0d9e admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:30.740205 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2d7d52-42c6-4b60-86db-7f544cbe0d9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:30.740487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2d7d52-42c6-4b60-86db-7f544cbe0d9e admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:30.741108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2d7d52-42c6-4b60-86db-7f544cbe0d9e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:30.743170 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2d7d52-42c6-4b60-86db-7f544cbe0d9e admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:30.762756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0aa55eb7-4edb-4dc3-bb32-85d46f857808 admin admin] The token cache is being invalidate because project 697abea4080645008cdac2b13c2baf71 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:30.767385 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0aa55eb7-4edb-4dc3-bb32-85d46f857808 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:30.767385 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0aa55eb7-4edb-4dc3-bb32-85d46f857808 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:30.767385 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:30.767875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1097/2188] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:30 2022] DELETE /identity/v3/projects/697abea4080645008cdac2b13c2baf71 => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:47:30.771533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa55eb7-4edb-4dc3-bb32-85d46f857808 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:30.772964 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0aa55eb7-4edb-4dc3-bb32-85d46f857808 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:30.835327 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa55eb7-4edb-4dc3-bb32-85d46f857808 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:30.840218 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e4316c-aa1b-4af7-84e9-9fcf7671377e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:30.840842 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e4316c-aa1b-4af7-84e9-9fcf7671377e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:30.841119 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e4316c-aa1b-4af7-84e9-9fcf7671377e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:30.841390 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e4316c-aa1b-4af7-84e9-9fcf7671377e admin admin] PATH_INFO: `/v3/users/5dc0c4d258fd4ac498ecea94ae1ecdd8` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:30.842497 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e4316c-aa1b-4af7-84e9-9fcf7671377e admin admin] RBAC: Authorizing `identity:delete_user(user_id=5dc0c4d258fd4ac498ecea94ae1ecdd8)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:30.843263 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1092/2189] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:30 2022] POST /identity/v3/users => generated 293 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:30.844682 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e4316c-aa1b-4af7-84e9-9fcf7671377e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:30.847292 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2d7d52-42c6-4b60-86db-7f544cbe0d9e admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:30.848917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ed2d7d52-42c6-4b60-86db-7f544cbe0d9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:30.861388 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2d7d52-42c6-4b60-86db-7f544cbe0d9e admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:30.866120 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce7dbe7-3695-440f-99c2-97500678f982 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:30.866745 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce7dbe7-3695-440f-99c2-97500678f982 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:30.867026 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce7dbe7-3695-440f-99c2-97500678f982 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:30.867296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce7dbe7-3695-440f-99c2-97500678f982 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:30.867939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce7dbe7-3695-440f-99c2-97500678f982 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:30.869803 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce7dbe7-3695-440f-99c2-97500678f982 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:30.880465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1093/2190] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:30 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:47:30.884405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce7dbe7-3695-440f-99c2-97500678f982 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:30.894078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce7dbe7-3695-440f-99c2-97500678f982 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:30.896428 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=93,keystone:DELETE=2,keystone:INSERT=5 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:30.900421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e73914d-c7be-408b-bf35-cf433b2f2f93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:30.900989 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e73914d-c7be-408b-bf35-cf433b2f2f93 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:30.905091 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e73914d-c7be-408b-bf35-cf433b2f2f93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:30.905091 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e73914d-c7be-408b-bf35-cf433b2f2f93 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:30.905091 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e73914d-c7be-408b-bf35-cf433b2f2f93 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:30.905091 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e73914d-c7be-408b-bf35-cf433b2f2f93 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:30.914228 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1094/2191] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:30.918192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e73914d-c7be-408b-bf35-cf433b2f2f93 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:30.928250 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e73914d-c7be-408b-bf35-cf433b2f2f93 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:30.933061 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61235248-6667-4815-9f3b-8c9bb9c55b5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:30.933683 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61235248-6667-4815-9f3b-8c9bb9c55b5d admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:30.933982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61235248-6667-4815-9f3b-8c9bb9c55b5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:30.934240 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61235248-6667-4815-9f3b-8c9bb9c55b5d admin admin] PATH_INFO: `/v3/projects/76a9523355aa44049fe91113df562bc1/users/3f16a7dfcba7411cb3aea401f2a6ae10/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:30.953783 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61235248-6667-4815-9f3b-8c9bb9c55b5d admin admin] RBAC: Authorizing `identity:create_grant(project_id=76a9523355aa44049fe91113df562bc1, user_id=3f16a7dfcba7411cb3aea401f2a6ae10, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:30.953783 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61235248-6667-4815-9f3b-8c9bb9c55b5d admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:31.041534 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-75e4316c-aa1b-4af7-84e9-9fcf7671377e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5dc0c4d258fd4ac498ecea94ae1ecdd8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:31.042278 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-75e4316c-aa1b-4af7-84e9-9fcf7671377e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5dc0c4d258fd4ac498ecea94ae1ecdd8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:31.042961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-75e4316c-aa1b-4af7-84e9-9fcf7671377e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5dc0c4d258fd4ac498ecea94ae1ecdd8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:31.050814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-75e4316c-aa1b-4af7-84e9-9fcf7671377e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5dc0c4d258fd4ac498ecea94ae1ecdd8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:31.074205 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1095/2192] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:30 2022] PUT /identity/v3/projects/76a9523355aa44049fe91113df562bc1/users/3f16a7dfcba7411cb3aea401f2a6ae10/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:47:31.084438 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455893dc-c997-45f2-9fc7-a8111a41c6bd None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:31.084826 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455893dc-c997-45f2-9fc7-a8111a41c6bd None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:31.085174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455893dc-c997-45f2-9fc7-a8111a41c6bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:31.113593 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-455893dc-c997-45f2-9fc7-a8111a41c6bd None None] MFA Rules not processed for user `3f16a7dfcba7411cb3aea401f2a6ae10`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:31.115298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-455893dc-c997-45f2-9fc7-a8111a41c6bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:31.128693 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-75e4316c-aa1b-4af7-84e9-9fcf7671377e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5dc0c4d258fd4ac498ecea94ae1ecdd8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:31.154423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1098/2193] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:30 2022] DELETE /identity/v3/users/5dc0c4d258fd4ac498ecea94ae1ecdd8 => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:47:31.190075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:31.204457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1096/2194] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:31 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:31.216455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e4316c-aa1b-4af7-84e9-9fcf7671377e admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:31.217894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-75e4316c-aa1b-4af7-84e9-9fcf7671377e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:31.274548 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e4316c-aa1b-4af7-84e9-9fcf7671377e admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:31.278262 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b092942e-4cef-4172-a395-aa29b3186806 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:31.279446 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b092942e-4cef-4172-a395-aa29b3186806 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:31.279731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b092942e-4cef-4172-a395-aa29b3186806 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:31.280006 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b092942e-4cef-4172-a395-aa29b3186806 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:31.339411 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b092942e-4cef-4172-a395-aa29b3186806 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:31.346351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b092942e-4cef-4172-a395-aa29b3186806 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:31.348650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:31.352620 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1099/2195] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:31 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:31.377267 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:31.566568 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455893dc-c997-45f2-9fc7-a8111a41c6bd None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:31.568077 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-455893dc-c997-45f2-9fc7-a8111a41c6bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:31.635714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455893dc-c997-45f2-9fc7-a8111a41c6bd None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:31.643279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=93,keystone:INSERT=1,keystone:DELETE=6 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:31.674904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:31.692424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9c2189-4875-40a8-8caf-c97c03cce8f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:31.693050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9c2189-4875-40a8-8caf-c97c03cce8f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:31.693330 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9c2189-4875-40a8-8caf-c97c03cce8f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:31.693628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9c2189-4875-40a8-8caf-c97c03cce8f1 admin admin] PATH_INFO: `/v3/projects/68a56179fff34e0bac65384b683e7b83` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:31.694789 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9c2189-4875-40a8-8caf-c97c03cce8f1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=68a56179fff34e0bac65384b683e7b83)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:31.696708 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9c2189-4875-40a8-8caf-c97c03cce8f1 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:31.791510 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-1d9c2189-4875-40a8-8caf-c97c03cce8f1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '68a56179fff34e0bac65384b683e7b83', 'request_id': 'req-1d9c2189-4875-40a8-8caf-c97c03cce8f1'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:31.797405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-1d9c2189-4875-40a8-8caf-c97c03cce8f1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '68a56179fff34e0bac65384b683e7b83', 'request_id': 'req-1d9c2189-4875-40a8-8caf-c97c03cce8f1'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:31.803491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-1d9c2189-4875-40a8-8caf-c97c03cce8f1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '68a56179fff34e0bac65384b683e7b83', 'request_id': 'req-1d9c2189-4875-40a8-8caf-c97c03cce8f1'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:31.889341 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-1d9c2189-4875-40a8-8caf-c97c03cce8f1 admin admin] The token cache is being invalidate because project 68a56179fff34e0bac65384b683e7b83 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:31.889633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-1d9c2189-4875-40a8-8caf-c97c03cce8f1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:31.890262 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-1d9c2189-4875-40a8-8caf-c97c03cce8f1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:31.891775 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1097/2196] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:31 2022] DELETE /identity/v3/projects/68a56179fff34e0bac65384b683e7b83 => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:31.897491 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafef024-36a5-45d8-af4e-b23113317678 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:31.897793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafef024-36a5-45d8-af4e-b23113317678 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:31.898052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafef024-36a5-45d8-af4e-b23113317678 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:31.917278 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-cafef024-36a5-45d8-af4e-b23113317678 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:31.918471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-cafef024-36a5-45d8-af4e-b23113317678 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:31.981367 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1100/2197] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:31.990191 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757927f7-2180-4a46-bae0-3b2a1d1fa128 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:31.990474 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757927f7-2180-4a46-bae0-3b2a1d1fa128 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:31.990923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757927f7-2180-4a46-bae0-3b2a1d1fa128 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:32.015678 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-757927f7-2180-4a46-bae0-3b2a1d1fa128 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:32.016911 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-757927f7-2180-4a46-bae0-3b2a1d1fa128 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:32.026552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1098/2198] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:32.031636 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafef024-36a5-45d8-af4e-b23113317678 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:32.041935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafef024-36a5-45d8-af4e-b23113317678 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:32.047569 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3b5bd8-ab53-4a22-958c-c094ac1b6a14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:32.048173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3b5bd8-ab53-4a22-958c-c094ac1b6a14 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:32.048455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3b5bd8-ab53-4a22-958c-c094ac1b6a14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:32.048728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3b5bd8-ab53-4a22-958c-c094ac1b6a14 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:32.049440 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3b5bd8-ab53-4a22-958c-c094ac1b6a14 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:47:32.049740 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3b5bd8-ab53-4a22-958c-c094ac1b6a14 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:32.051361 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3b5bd8-ab53-4a22-958c-c094ac1b6a14 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:32.067283 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1101/2199] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:47:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:47:32.071643 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757927f7-2180-4a46-bae0-3b2a1d1fa128 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:32.078710 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:32.082381 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757927f7-2180-4a46-bae0-3b2a1d1fa128 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:32.087118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a525dfd9-8e13-474c-ae7d-586e33070607 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:32.087707 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a525dfd9-8e13-474c-ae7d-586e33070607 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:32.087981 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a525dfd9-8e13-474c-ae7d-586e33070607 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:32.088326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a525dfd9-8e13-474c-ae7d-586e33070607 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:32.089054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a525dfd9-8e13-474c-ae7d-586e33070607 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:32.091377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a525dfd9-8e13-474c-ae7d-586e33070607 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:32.131122 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:32.178761 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1099/2200] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:32 2022] POST /identity/v3/projects => generated 373 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:32.182929 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3b5bd8-ab53-4a22-958c-c094ac1b6a14 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:32.193849 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3b5bd8-ab53-4a22-958c-c094ac1b6a14 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:32.270878 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c55fd96-364b-44e1-bfb4-c215c53d3bdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:32.271507 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c55fd96-364b-44e1-bfb4-c215c53d3bdd admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:32.271853 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c55fd96-364b-44e1-bfb4-c215c53d3bdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:32.272189 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c55fd96-364b-44e1-bfb4-c215c53d3bdd admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:32.272891 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c55fd96-364b-44e1-bfb4-c215c53d3bdd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:32.275020 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c55fd96-364b-44e1-bfb4-c215c53d3bdd admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:32.376184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1102/2201] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:32 2022] POST /identity/v3/users => generated 287 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:32.380515 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a525dfd9-8e13-474c-ae7d-586e33070607 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:32.391288 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a525dfd9-8e13-474c-ae7d-586e33070607 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:32.395975 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e68b70-f6c4-4aec-b45d-4e738eb044a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:32.396595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e68b70-f6c4-4aec-b45d-4e738eb044a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:32.396909 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e68b70-f6c4-4aec-b45d-4e738eb044a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:32.397153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e68b70-f6c4-4aec-b45d-4e738eb044a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:32.397894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e68b70-f6c4-4aec-b45d-4e738eb044a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:32.399543 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e68b70-f6c4-4aec-b45d-4e738eb044a0 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:32.410290 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1100/2202] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:32 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:32.414439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c55fd96-364b-44e1-bfb4-c215c53d3bdd admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:32.424920 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c55fd96-364b-44e1-bfb4-c215c53d3bdd admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:32.429780 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a503eb2f-9f8c-41ec-8f50-1a00e32e0818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:32.432720 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a503eb2f-9f8c-41ec-8f50-1a00e32e0818 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:32.432720 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a503eb2f-9f8c-41ec-8f50-1a00e32e0818 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:32.432720 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a503eb2f-9f8c-41ec-8f50-1a00e32e0818 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:32.432720 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a503eb2f-9f8c-41ec-8f50-1a00e32e0818 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:32.433185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a503eb2f-9f8c-41ec-8f50-1a00e32e0818 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:32.445028 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1103/2203] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:47:32.449644 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e68b70-f6c4-4aec-b45d-4e738eb044a0 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:32.460219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e68b70-f6c4-4aec-b45d-4e738eb044a0 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:32.465058 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3bea57-e547-4eb3-87b5-4d6b41df66e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:32.465702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3bea57-e547-4eb3-87b5-4d6b41df66e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:32.466726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3bea57-e547-4eb3-87b5-4d6b41df66e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:32.467005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3bea57-e547-4eb3-87b5-4d6b41df66e0 admin admin] PATH_INFO: `/v3/projects/2829024de36b4ad89d3d1173a7cc3c37/users/0cb262dfc2014569afe6f2c5498f6be3/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:32.482197 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3bea57-e547-4eb3-87b5-4d6b41df66e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2829024de36b4ad89d3d1173a7cc3c37, user_id=0cb262dfc2014569afe6f2c5498f6be3, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:32.490268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3bea57-e547-4eb3-87b5-4d6b41df66e0 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:32.531567 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:DELETE=8,keystone:SELECT=90,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:32.567023 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1101/2204] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:32 2022] PUT /identity/v3/projects/2829024de36b4ad89d3d1173a7cc3c37/users/0cb262dfc2014569afe6f2c5498f6be3/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:47:32.577694 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe16284-aa4d-4c68-becf-a6b211ec80e9 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:32.578002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe16284-aa4d-4c68-becf-a6b211ec80e9 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:32.578275 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe16284-aa4d-4c68-becf-a6b211ec80e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:32.606183 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-cfe16284-aa4d-4c68-becf-a6b211ec80e9 None None] MFA Rules not processed for user `0cb262dfc2014569afe6f2c5498f6be3`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:32.607457 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-cfe16284-aa4d-4c68-becf-a6b211ec80e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:32.671822 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1104/2205] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:32 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:32.682275 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3bea57-e547-4eb3-87b5-4d6b41df66e0 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:32.683967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-6f3bea57-e547-4eb3-87b5-4d6b41df66e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:32.744763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3bea57-e547-4eb3-87b5-4d6b41df66e0 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:32.748938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f06bb4-d5b8-440e-bad9-19cddf56200a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:32.749542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f06bb4-d5b8-440e-bad9-19cddf56200a service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:32.749819 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f06bb4-d5b8-440e-bad9-19cddf56200a service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:32.750089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f06bb4-d5b8-440e-bad9-19cddf56200a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:32.758199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f06bb4-d5b8-440e-bad9-19cddf56200a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:32.759710 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f06bb4-d5b8-440e-bad9-19cddf56200a service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:32.769389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1102/2206] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:32 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:33.020772 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=81,keystone:INSERT=3,keystone:DELETE=16 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:33.894935 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:34.700907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:34.850933 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:45.124994 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=56,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:45.210547 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=21 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:48.148312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe16284-aa4d-4c68-becf-a6b211ec80e9 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:48.152044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-cfe16284-aa4d-4c68-becf-a6b211ec80e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:48.250617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe16284-aa4d-4c68-becf-a6b211ec80e9 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:48.255162 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c788fa4-f08b-47d9-b9ab-97e54d25827d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:48.255792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c788fa4-f08b-47d9-b9ab-97e54d25827d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:48.256070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c788fa4-f08b-47d9-b9ab-97e54d25827d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:48.256340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c788fa4-f08b-47d9-b9ab-97e54d25827d admin admin] PATH_INFO: `/v3/users/a0ddd0819da6453fb289ac068bdde657` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:48.257456 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c788fa4-f08b-47d9-b9ab-97e54d25827d admin admin] RBAC: Authorizing `identity:delete_user(user_id=a0ddd0819da6453fb289ac068bdde657)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:48.259403 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c788fa4-f08b-47d9-b9ab-97e54d25827d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:48.479267 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1c788fa4-f08b-47d9-b9ab-97e54d25827d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a0ddd0819da6453fb289ac068bdde657'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:48.479931 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1c788fa4-f08b-47d9-b9ab-97e54d25827d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a0ddd0819da6453fb289ac068bdde657'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:48.480557 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1c788fa4-f08b-47d9-b9ab-97e54d25827d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a0ddd0819da6453fb289ac068bdde657'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:48.490949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1c788fa4-f08b-47d9-b9ab-97e54d25827d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a0ddd0819da6453fb289ac068bdde657'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:48.585097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1c788fa4-f08b-47d9-b9ab-97e54d25827d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a0ddd0819da6453fb289ac068bdde657'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:48.615190 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1105/2207] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:48 2022] DELETE /identity/v3/users/a0ddd0819da6453fb289ac068bdde657 => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:48.620002 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860f249d-6412-4a33-bb9e-b82ec3cc6b44 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:48.620542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860f249d-6412-4a33-bb9e-b82ec3cc6b44 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:48.620829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860f249d-6412-4a33-bb9e-b82ec3cc6b44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:48.639488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-860f249d-6412-4a33-bb9e-b82ec3cc6b44 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:48.640683 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-860f249d-6412-4a33-bb9e-b82ec3cc6b44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:48.699918 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1103/2208] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:48.709767 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c788fa4-f08b-47d9-b9ab-97e54d25827d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:48.711187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1c788fa4-f08b-47d9-b9ab-97e54d25827d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:48.774095 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c788fa4-f08b-47d9-b9ab-97e54d25827d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:48.777899 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c819fdd-04d8-41a2-ad09-1759eef8d165 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:48.778669 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c819fdd-04d8-41a2-ad09-1759eef8d165 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:48.778952 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c819fdd-04d8-41a2-ad09-1759eef8d165 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:48.779227 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c819fdd-04d8-41a2-ad09-1759eef8d165 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:48.790861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c819fdd-04d8-41a2-ad09-1759eef8d165 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:48.792646 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c819fdd-04d8-41a2-ad09-1759eef8d165 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:48.803058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1106/2209] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:48.865008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860f249d-6412-4a33-bb9e-b82ec3cc6b44 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:48.866434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-860f249d-6412-4a33-bb9e-b82ec3cc6b44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:48.878891 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860f249d-6412-4a33-bb9e-b82ec3cc6b44 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:48.883639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001ee8e7-0266-4c7f-86ec-37355880373f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:48.884271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001ee8e7-0266-4c7f-86ec-37355880373f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:48.884554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001ee8e7-0266-4c7f-86ec-37355880373f admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:48.884815 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001ee8e7-0266-4c7f-86ec-37355880373f admin admin] PATH_INFO: `/v3/users/3f16a7dfcba7411cb3aea401f2a6ae10` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:48.885941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001ee8e7-0266-4c7f-86ec-37355880373f admin admin] RBAC: Authorizing `identity:delete_user(user_id=3f16a7dfcba7411cb3aea401f2a6ae10)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:48.888055 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001ee8e7-0266-4c7f-86ec-37355880373f admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:49.075682 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c819fdd-04d8-41a2-ad09-1759eef8d165 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:49.078001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7c819fdd-04d8-41a2-ad09-1759eef8d165 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:49.098077 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c819fdd-04d8-41a2-ad09-1759eef8d165 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:49.103656 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff207059-1fed-49b8-a95f-a8f285fcc4be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:49.104558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff207059-1fed-49b8-a95f-a8f285fcc4be admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:49.104936 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff207059-1fed-49b8-a95f-a8f285fcc4be admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:49.105324 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff207059-1fed-49b8-a95f-a8f285fcc4be admin admin] PATH_INFO: `/v3/users/ed9d2408c78d426d8cbfcfd7839da69b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:49.106540 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff207059-1fed-49b8-a95f-a8f285fcc4be admin admin] RBAC: Authorizing `identity:delete_user(user_id=ed9d2408c78d426d8cbfcfd7839da69b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:49.108613 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff207059-1fed-49b8-a95f-a8f285fcc4be admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:49.130255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:49.130467 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-001ee8e7-0266-4c7f-86ec-37355880373f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3f16a7dfcba7411cb3aea401f2a6ae10'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:49.145688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-001ee8e7-0266-4c7f-86ec-37355880373f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3f16a7dfcba7411cb3aea401f2a6ae10'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:49.146711 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-001ee8e7-0266-4c7f-86ec-37355880373f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3f16a7dfcba7411cb3aea401f2a6ae10'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:49.221577 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-001ee8e7-0266-4c7f-86ec-37355880373f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3f16a7dfcba7411cb3aea401f2a6ae10'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:49.231074 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-001ee8e7-0266-4c7f-86ec-37355880373f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3f16a7dfcba7411cb3aea401f2a6ae10'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:49.236599 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1104/2210] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:48 2022] DELETE /identity/v3/users/3f16a7dfcba7411cb3aea401f2a6ae10 => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:49.241275 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7453f350-44f0-4499-ad44-9e4489b5e47d None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:49.241637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7453f350-44f0-4499-ad44-9e4489b5e47d None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:49.242005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7453f350-44f0-4499-ad44-9e4489b5e47d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:49.267241 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-7453f350-44f0-4499-ad44-9e4489b5e47d None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:49.268455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7453f350-44f0-4499-ad44-9e4489b5e47d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:49.331631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ff207059-1fed-49b8-a95f-a8f285fcc4be admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ed9d2408c78d426d8cbfcfd7839da69b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:49.332861 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ff207059-1fed-49b8-a95f-a8f285fcc4be admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ed9d2408c78d426d8cbfcfd7839da69b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:49.333541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ff207059-1fed-49b8-a95f-a8f285fcc4be admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ed9d2408c78d426d8cbfcfd7839da69b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:49.334247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1105/2211] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:47:49.337493 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7453f350-44f0-4499-ad44-9e4489b5e47d None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:49.338559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7453f350-44f0-4499-ad44-9e4489b5e47d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:49.340918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ff207059-1fed-49b8-a95f-a8f285fcc4be admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ed9d2408c78d426d8cbfcfd7839da69b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:49.350830 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7453f350-44f0-4499-ad44-9e4489b5e47d None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:49.355556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d963bc5f-91d8-4928-bda2-f65249459d53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:49.356201 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d963bc5f-91d8-4928-bda2-f65249459d53 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:49.356474 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d963bc5f-91d8-4928-bda2-f65249459d53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:49.356774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d963bc5f-91d8-4928-bda2-f65249459d53 admin admin] PATH_INFO: `/v3/projects/b39a7111b78845219da0f1c53007fc4a` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:49.357999 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d963bc5f-91d8-4928-bda2-f65249459d53 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b39a7111b78845219da0f1c53007fc4a)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:49.359998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d963bc5f-91d8-4928-bda2-f65249459d53 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:49.368142 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:49.437121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ff207059-1fed-49b8-a95f-a8f285fcc4be admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ed9d2408c78d426d8cbfcfd7839da69b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:49.456636 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1107/2212] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:49 2022] DELETE /identity/v3/users/ed9d2408c78d426d8cbfcfd7839da69b => generated 0 bytes in 382 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:47:49.457650 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff207059-1fed-49b8-a95f-a8f285fcc4be admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:49.459183 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ff207059-1fed-49b8-a95f-a8f285fcc4be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:49.468494 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-d963bc5f-91d8-4928-bda2-f65249459d53 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b39a7111b78845219da0f1c53007fc4a', 'request_id': 'req-d963bc5f-91d8-4928-bda2-f65249459d53'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:49.474361 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-d963bc5f-91d8-4928-bda2-f65249459d53 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b39a7111b78845219da0f1c53007fc4a', 'request_id': 'req-d963bc5f-91d8-4928-bda2-f65249459d53'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:49.481032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-d963bc5f-91d8-4928-bda2-f65249459d53 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b39a7111b78845219da0f1c53007fc4a', 'request_id': 'req-d963bc5f-91d8-4928-bda2-f65249459d53'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:49.517096 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff207059-1fed-49b8-a95f-a8f285fcc4be admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:49.520763 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2346d12c-e45d-460f-87b0-19a15a10fb43 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:49.521405 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2346d12c-e45d-460f-87b0-19a15a10fb43 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:49.521688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2346d12c-e45d-460f-87b0-19a15a10fb43 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:49.521964 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2346d12c-e45d-460f-87b0-19a15a10fb43 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:49.523893 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2346d12c-e45d-460f-87b0-19a15a10fb43 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:49.552828 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-d963bc5f-91d8-4928-bda2-f65249459d53 admin admin] The token cache is being invalidate because project b39a7111b78845219da0f1c53007fc4a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:49.553050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-d963bc5f-91d8-4928-bda2-f65249459d53 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:49.553689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-d963bc5f-91d8-4928-bda2-f65249459d53 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:49.555424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1106/2213] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:49 2022] DELETE /identity/v3/projects/b39a7111b78845219da0f1c53007fc4a => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:49.556421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae763d6-6f1e-4d97-8699-b04b6d779494 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:49.556705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae763d6-6f1e-4d97-8699-b04b6d779494 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:49.556982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae763d6-6f1e-4d97-8699-b04b6d779494 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:49.579453 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-9ae763d6-6f1e-4d97-8699-b04b6d779494 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:49.580712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-9ae763d6-6f1e-4d97-8699-b04b6d779494 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:49.588729 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1107/2214] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:49.590001 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2aa6fe-7dce-4fdb-9521-83f45873d3ed None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:49.590283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2aa6fe-7dce-4fdb-9521-83f45873d3ed None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:49.590484 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2346d12c-e45d-460f-87b0-19a15a10fb43 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:49.590612 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2aa6fe-7dce-4fdb-9521-83f45873d3ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:49.592254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2346d12c-e45d-460f-87b0-19a15a10fb43 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:49.603113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1108/2215] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:49.604054 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2346d12c-e45d-460f-87b0-19a15a10fb43 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:49.612905 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-4b2aa6fe-7dce-4fdb-9521-83f45873d3ed None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:49.612905 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-4b2aa6fe-7dce-4fdb-9521-83f45873d3ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:49.613177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2346d12c-e45d-460f-87b0-19a15a10fb43 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:49.616867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3331a2-9673-4500-bbef-981d75528c79 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:49.617553 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3331a2-9673-4500-bbef-981d75528c79 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:49.617917 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3331a2-9673-4500-bbef-981d75528c79 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:49.618265 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3331a2-9673-4500-bbef-981d75528c79 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:49.621919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1108/2216] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:49.629671 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3331a2-9673-4500-bbef-981d75528c79 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:49.630883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e869f8-d76b-483b-8bd6-7e447596f929 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:49.631361 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e869f8-d76b-483b-8bd6-7e447596f929 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:49.631468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3331a2-9673-4500-bbef-981d75528c79 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:49.631764 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e869f8-d76b-483b-8bd6-7e447596f929 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:49.643611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1109/2217] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:49.654091 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-c3e869f8-d76b-483b-8bd6-7e447596f929 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:49.655387 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c3e869f8-d76b-483b-8bd6-7e447596f929 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:49.663660 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1109/2218] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:49.668233 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3331a2-9673-4500-bbef-981d75528c79 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:49.678613 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3331a2-9673-4500-bbef-981d75528c79 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:49.683504 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e739c41-1458-4a54-866d-a054c69e8a76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:49.684101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e739c41-1458-4a54-866d-a054c69e8a76 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:49.684374 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e739c41-1458-4a54-866d-a054c69e8a76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:49.684643 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e739c41-1458-4a54-866d-a054c69e8a76 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:49.685275 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e739c41-1458-4a54-866d-a054c69e8a76 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:47:49.685575 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e739c41-1458-4a54-866d-a054c69e8a76 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:49.687831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e739c41-1458-4a54-866d-a054c69e8a76 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:49.700201 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1110/2219] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:47:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:47:49.704794 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e869f8-d76b-483b-8bd6-7e447596f929 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:49.715667 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e869f8-d76b-483b-8bd6-7e447596f929 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:49.721285 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3538228-9e60-44f7-a64d-c9417c9681dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:49.721889 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3538228-9e60-44f7-a64d-c9417c9681dc admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:49.722164 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3538228-9e60-44f7-a64d-c9417c9681dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:49.722475 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3538228-9e60-44f7-a64d-c9417c9681dc admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:49.723174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3538228-9e60-44f7-a64d-c9417c9681dc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:49.725172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3538228-9e60-44f7-a64d-c9417c9681dc admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:49.907380 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e739c41-1458-4a54-866d-a054c69e8a76 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:49.907844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8e739c41-1458-4a54-866d-a054c69e8a76 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:49.920475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e739c41-1458-4a54-866d-a054c69e8a76 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:49.925550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0637441a-b0eb-4afd-b515-160fb3eb9b4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:49.926169 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0637441a-b0eb-4afd-b515-160fb3eb9b4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:49.926580 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0637441a-b0eb-4afd-b515-160fb3eb9b4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:49.926928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0637441a-b0eb-4afd-b515-160fb3eb9b4a admin admin] PATH_INFO: `/v3/users/6f80ef9b0f0440eeade62a9b6070db35` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:49.928032 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0637441a-b0eb-4afd-b515-160fb3eb9b4a admin admin] RBAC: Authorizing `identity:delete_user(user_id=6f80ef9b0f0440eeade62a9b6070db35)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:49.929938 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0637441a-b0eb-4afd-b515-160fb3eb9b4a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:49.942573 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:50.001743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1110/2220] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:49 2022] POST /identity/v3/projects => generated 363 bytes in 297 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:47:50.006244 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3538228-9e60-44f7-a64d-c9417c9681dc admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:50.077279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3538228-9e60-44f7-a64d-c9417c9681dc admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:50.081996 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4439502-f880-4245-b081-af1c6397c291 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:50.082686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4439502-f880-4245-b081-af1c6397c291 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:50.082970 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4439502-f880-4245-b081-af1c6397c291 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:50.083243 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4439502-f880-4245-b081-af1c6397c291 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:50.083901 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4439502-f880-4245-b081-af1c6397c291 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:50.086219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4439502-f880-4245-b081-af1c6397c291 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:50.151073 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0637441a-b0eb-4afd-b515-160fb3eb9b4a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6f80ef9b0f0440eeade62a9b6070db35'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:50.152432 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0637441a-b0eb-4afd-b515-160fb3eb9b4a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6f80ef9b0f0440eeade62a9b6070db35'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:50.153045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0637441a-b0eb-4afd-b515-160fb3eb9b4a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6f80ef9b0f0440eeade62a9b6070db35'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:50.161750 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0637441a-b0eb-4afd-b515-160fb3eb9b4a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6f80ef9b0f0440eeade62a9b6070db35'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:50.176948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1111/2221] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:50 2022] POST /identity/v3/users => generated 282 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:47:50.181744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4439502-f880-4245-b081-af1c6397c291 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:50.186508 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-f4439502-f880-4245-b081-af1c6397c291 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:50.200298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4439502-f880-4245-b081-af1c6397c291 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:50.205134 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0283b5d4-fab6-4e62-b7bb-55afc372f847 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:50.205736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0283b5d4-fab6-4e62-b7bb-55afc372f847 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:50.206017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0283b5d4-fab6-4e62-b7bb-55afc372f847 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:50.206487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0283b5d4-fab6-4e62-b7bb-55afc372f847 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:50.207811 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0283b5d4-fab6-4e62-b7bb-55afc372f847 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:50.211851 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0283b5d4-fab6-4e62-b7bb-55afc372f847 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:50.222218 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1112/2222] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:50 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:50.227949 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0283b5d4-fab6-4e62-b7bb-55afc372f847 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:50.238243 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0283b5d4-fab6-4e62-b7bb-55afc372f847 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:50.245428 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233d3812-27a3-4f97-906e-35b3f7ea35bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:50.246195 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233d3812-27a3-4f97-906e-35b3f7ea35bb admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:50.246580 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233d3812-27a3-4f97-906e-35b3f7ea35bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:50.246993 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233d3812-27a3-4f97-906e-35b3f7ea35bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:50.247712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233d3812-27a3-4f97-906e-35b3f7ea35bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:50.249596 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233d3812-27a3-4f97-906e-35b3f7ea35bb admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:50.261041 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1113/2223] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:50 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:47:50.265717 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233d3812-27a3-4f97-906e-35b3f7ea35bb admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:50.275471 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233d3812-27a3-4f97-906e-35b3f7ea35bb admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:50.280926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf49dfd-8df8-4a14-9e79-fb144b27e07a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:50.281730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf49dfd-8df8-4a14-9e79-fb144b27e07a admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:50.282055 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf49dfd-8df8-4a14-9e79-fb144b27e07a admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:50.283234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf49dfd-8df8-4a14-9e79-fb144b27e07a admin admin] PATH_INFO: `/v3/projects/13a92def01b54713a3acf8d15726c7d6/users/1c1e525815e9407a8f5d8c58b0b7d6fa/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:50.285942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0637441a-b0eb-4afd-b515-160fb3eb9b4a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6f80ef9b0f0440eeade62a9b6070db35'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:50.307148 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf49dfd-8df8-4a14-9e79-fb144b27e07a admin admin] RBAC: Authorizing `identity:create_grant(project_id=13a92def01b54713a3acf8d15726c7d6, user_id=1c1e525815e9407a8f5d8c58b0b7d6fa, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:50.310871 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf49dfd-8df8-4a14-9e79-fb144b27e07a admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:50.343502 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1111/2224] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:49 2022] DELETE /identity/v3/users/6f80ef9b0f0440eeade62a9b6070db35 => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:47:50.344842 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0637441a-b0eb-4afd-b515-160fb3eb9b4a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:50.347128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0637441a-b0eb-4afd-b515-160fb3eb9b4a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:50.388690 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=87,keystone:DELETE=7,keystone:INSERT=6 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:50.396561 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1114/2225] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:50 2022] PUT /identity/v3/projects/13a92def01b54713a3acf8d15726c7d6/users/1c1e525815e9407a8f5d8c58b0b7d6fa/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:47:50.397864 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf49dfd-8df8-4a14-9e79-fb144b27e07a admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:50.399629 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ecf49dfd-8df8-4a14-9e79-fb144b27e07a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:50.419676 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0637441a-b0eb-4afd-b515-160fb3eb9b4a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:50.466494 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf49dfd-8df8-4a14-9e79-fb144b27e07a admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:50.472221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b12791-281b-4f25-885d-61b786c28adc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:50.472870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b12791-281b-4f25-885d-61b786c28adc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:50.473155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b12791-281b-4f25-885d-61b786c28adc admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:50.473438 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b12791-281b-4f25-885d-61b786c28adc admin admin] PATH_INFO: `/v3/users/0cb262dfc2014569afe6f2c5498f6be3` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:50.474626 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b12791-281b-4f25-885d-61b786c28adc admin admin] RBAC: Authorizing `identity:delete_user(user_id=0cb262dfc2014569afe6f2c5498f6be3)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:50.476680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b12791-281b-4f25-885d-61b786c28adc admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:50.484974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ba893c-a01c-4695-ac5f-5aa148678f29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:50.484974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ba893c-a01c-4695-ac5f-5aa148678f29 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:50.484974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ba893c-a01c-4695-ac5f-5aa148678f29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:50.484974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ba893c-a01c-4695-ac5f-5aa148678f29 admin admin] PATH_INFO: `/v3/projects/76a9523355aa44049fe91113df562bc1` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:50.485325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ba893c-a01c-4695-ac5f-5aa148678f29 admin admin] RBAC: Authorizing `identity:delete_project(project_id=76a9523355aa44049fe91113df562bc1)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:50.485423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ba893c-a01c-4695-ac5f-5aa148678f29 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:50.738678 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-81ba893c-a01c-4695-ac5f-5aa148678f29 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '76a9523355aa44049fe91113df562bc1', 'request_id': 'req-81ba893c-a01c-4695-ac5f-5aa148678f29'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:50.740035 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-59b12791-281b-4f25-885d-61b786c28adc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0cb262dfc2014569afe6f2c5498f6be3'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:50.744814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-81ba893c-a01c-4695-ac5f-5aa148678f29 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '76a9523355aa44049fe91113df562bc1', 'request_id': 'req-81ba893c-a01c-4695-ac5f-5aa148678f29'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:50.751520 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-81ba893c-a01c-4695-ac5f-5aa148678f29 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '76a9523355aa44049fe91113df562bc1', 'request_id': 'req-81ba893c-a01c-4695-ac5f-5aa148678f29'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:50.752989 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-59b12791-281b-4f25-885d-61b786c28adc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0cb262dfc2014569afe6f2c5498f6be3'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:50.753685 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-59b12791-281b-4f25-885d-61b786c28adc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0cb262dfc2014569afe6f2c5498f6be3'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:50.790822 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:50.824380 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-81ba893c-a01c-4695-ac5f-5aa148678f29 admin admin] The token cache is being invalidate because project 76a9523355aa44049fe91113df562bc1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:50.824699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-81ba893c-a01c-4695-ac5f-5aa148678f29 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:50.825376 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-81ba893c-a01c-4695-ac5f-5aa148678f29 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:50.826928 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1112/2226] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:50 2022] DELETE /identity/v3/projects/76a9523355aa44049fe91113df562bc1 => generated 0 bytes in 482 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:50.828714 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e9d317-006f-4052-867e-f145974b0419 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:50.829000 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e9d317-006f-4052-867e-f145974b0419 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:50.829274 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e9d317-006f-4052-867e-f145974b0419 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:50.848736 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-a6e9d317-006f-4052-867e-f145974b0419 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:50.849943 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a6e9d317-006f-4052-867e-f145974b0419 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:50.899742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-59b12791-281b-4f25-885d-61b786c28adc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0cb262dfc2014569afe6f2c5498f6be3'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:50.907222 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-59b12791-281b-4f25-885d-61b786c28adc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0cb262dfc2014569afe6f2c5498f6be3'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:50.909545 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1113/2227] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:50.909545 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c4e09-e3ac-4f7a-865b-9b4c43720653 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:50.909545 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c4e09-e3ac-4f7a-865b-9b4c43720653 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:50.909545 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c4e09-e3ac-4f7a-865b-9b4c43720653 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:50.916032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1115/2228] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:50 2022] DELETE /identity/v3/users/0cb262dfc2014569afe6f2c5498f6be3 => generated 0 bytes in 519 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:50.919987 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b12791-281b-4f25-885d-61b786c28adc admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:50.921702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-59b12791-281b-4f25-885d-61b786c28adc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:50.940919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d49c4e09-e3ac-4f7a-865b-9b4c43720653 None None] MFA Rules not processed for user `1c1e525815e9407a8f5d8c58b0b7d6fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:50.942129 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d49c4e09-e3ac-4f7a-865b-9b4c43720653 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:50.988975 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b12791-281b-4f25-885d-61b786c28adc admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:50.994298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aec2939-b0dd-43a4-8911-4f991d554bda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:50.994984 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aec2939-b0dd-43a4-8911-4f991d554bda admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:50.995316 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aec2939-b0dd-43a4-8911-4f991d554bda admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:50.995589 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aec2939-b0dd-43a4-8911-4f991d554bda admin admin] PATH_INFO: `/v3/projects/584de4573964403fb0bace83b40e0761` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:50.996690 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aec2939-b0dd-43a4-8911-4f991d554bda admin admin] RBAC: Authorizing `identity:delete_project(project_id=584de4573964403fb0bace83b40e0761)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:50.998671 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aec2939-b0dd-43a4-8911-4f991d554bda admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:51.006990 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1114/2229] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:50 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:51.008227 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bd6a56-892a-496f-8b22-363ebe22d738 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:51.008624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bd6a56-892a-496f-8b22-363ebe22d738 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:51.008926 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bd6a56-892a-496f-8b22-363ebe22d738 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:51.028848 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e5bd6a56-892a-496f-8b22-363ebe22d738 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:51.030723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e5bd6a56-892a-496f-8b22-363ebe22d738 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:51.038938 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1115/2230] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:51.039876 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bd6a56-892a-496f-8b22-363ebe22d738 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:51.041093 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e5bd6a56-892a-496f-8b22-363ebe22d738 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:51.097972 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bd6a56-892a-496f-8b22-363ebe22d738 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:51.101608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29cccd7b-31ea-4fae-a83e-5960c6700239 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:51.102197 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cccd7b-31ea-4fae-a83e-5960c6700239 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:51.102487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cccd7b-31ea-4fae-a83e-5960c6700239 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:51.102778 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cccd7b-31ea-4fae-a83e-5960c6700239 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:51.111945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29cccd7b-31ea-4fae-a83e-5960c6700239 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:51.113356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29cccd7b-31ea-4fae-a83e-5960c6700239 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:51.123203 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1116/2231] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:51.125096 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91a700b-fd0d-489a-ab40-a6804c89ef36 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:51.125543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91a700b-fd0d-489a-ab40-a6804c89ef36 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:51.125934 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91a700b-fd0d-489a-ab40-a6804c89ef36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:51.132504 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5aec2939-b0dd-43a4-8911-4f991d554bda admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '584de4573964403fb0bace83b40e0761', 'request_id': 'req-5aec2939-b0dd-43a4-8911-4f991d554bda'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:51.138979 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5aec2939-b0dd-43a4-8911-4f991d554bda admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '584de4573964403fb0bace83b40e0761', 'request_id': 'req-5aec2939-b0dd-43a4-8911-4f991d554bda'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:51.144660 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f91a700b-fd0d-489a-ab40-a6804c89ef36 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:51.145761 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5aec2939-b0dd-43a4-8911-4f991d554bda admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '584de4573964403fb0bace83b40e0761', 'request_id': 'req-5aec2939-b0dd-43a4-8911-4f991d554bda'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:51.145907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f91a700b-fd0d-489a-ab40-a6804c89ef36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:51.182725 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:51.207174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1117/2232] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:51.208036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91a700b-fd0d-489a-ab40-a6804c89ef36 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:51.209349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f91a700b-fd0d-489a-ab40-a6804c89ef36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:51.218811 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5aec2939-b0dd-43a4-8911-4f991d554bda admin admin] The token cache is being invalidate because project 584de4573964403fb0bace83b40e0761 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:51.226885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5aec2939-b0dd-43a4-8911-4f991d554bda admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:51.226885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-5aec2939-b0dd-43a4-8911-4f991d554bda admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:51.226885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1116/2233] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:50 2022] DELETE /identity/v3/projects/584de4573964403fb0bace83b40e0761 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:51.226885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b15221-a066-448f-8e5f-8935048de80d None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:51.226885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b15221-a066-448f-8e5f-8935048de80d None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:51.226885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b15221-a066-448f-8e5f-8935048de80d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:51.244005 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-d2b15221-a066-448f-8e5f-8935048de80d None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:51.245211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d2b15221-a066-448f-8e5f-8935048de80d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:51.253712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1117/2234] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:51.254549 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b15221-a066-448f-8e5f-8935048de80d None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:51.255801 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d2b15221-a066-448f-8e5f-8935048de80d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:51.273113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91a700b-fd0d-489a-ab40-a6804c89ef36 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:51.278121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6864f80-ae34-4f66-8a8c-40f78d20ee02 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:51.278121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6864f80-ae34-4f66-8a8c-40f78d20ee02 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:51.278121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6864f80-ae34-4f66-8a8c-40f78d20ee02 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:51.278121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6864f80-ae34-4f66-8a8c-40f78d20ee02 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:51.279666 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d6864f80-ae34-4f66-8a8c-40f78d20ee02 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:51.315483 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b15221-a066-448f-8e5f-8935048de80d None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:51.319431 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d9d7fe-35ab-44a2-bbbb-11242cf2ea73 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:51.320637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d9d7fe-35ab-44a2-bbbb-11242cf2ea73 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:51.320927 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d9d7fe-35ab-44a2-bbbb-11242cf2ea73 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:51.321202 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d9d7fe-35ab-44a2-bbbb-11242cf2ea73 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:51.323672 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-73d9d7fe-35ab-44a2-bbbb-11242cf2ea73 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:51.338036 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d9d7fe-35ab-44a2-bbbb-11242cf2ea73 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:51.339597 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d9d7fe-35ab-44a2-bbbb-11242cf2ea73 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:51.342264 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6864f80-ae34-4f66-8a8c-40f78d20ee02 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:51.344039 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6864f80-ae34-4f66-8a8c-40f78d20ee02 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:51.354509 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1118/2235] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:51 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:51.355388 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6864f80-ae34-4f66-8a8c-40f78d20ee02 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:51.356712 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d6864f80-ae34-4f66-8a8c-40f78d20ee02 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:51.361243 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1118/2236] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:51.362827 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d9d7fe-35ab-44a2-bbbb-11242cf2ea73 service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:51.372134 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6864f80-ae34-4f66-8a8c-40f78d20ee02 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:51.379938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d9d7fe-35ab-44a2-bbbb-11242cf2ea73 service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:51.381894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c5fdd7-d6a7-49c0-8c8d-80793de01bff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:51.382621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c5fdd7-d6a7-49c0-8c8d-80793de01bff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:51.382835 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c5fdd7-d6a7-49c0-8c8d-80793de01bff admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:51.383250 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c5fdd7-d6a7-49c0-8c8d-80793de01bff admin admin] PATH_INFO: `/v3/users/c9ea2c12e28a4df8b4335522cc60b21d` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:51.384362 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c5fdd7-d6a7-49c0-8c8d-80793de01bff admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9ea2c12e28a4df8b4335522cc60b21d)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:51.386364 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c5fdd7-d6a7-49c0-8c8d-80793de01bff admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:51.388580 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe61192-ebd3-4c31-bbb3-e4d0f97dc449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:51.390089 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:51.391629 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe61192-ebd3-4c31-bbb3-e4d0f97dc449 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:51.392124 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe61192-ebd3-4c31-bbb3-e4d0f97dc449 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:51.392570 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe61192-ebd3-4c31-bbb3-e4d0f97dc449 admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:51.399268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe61192-ebd3-4c31-bbb3-e4d0f97dc449 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:47:51.399268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe61192-ebd3-4c31-bbb3-e4d0f97dc449 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:51.399268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe61192-ebd3-4c31-bbb3-e4d0f97dc449 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:51.417001 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1119/2237] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:47:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:47:51.421659 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe61192-ebd3-4c31-bbb3-e4d0f97dc449 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:51.435563 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe61192-ebd3-4c31-bbb3-e4d0f97dc449 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:51.441611 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469675c8-35c2-486e-a932-0403c49cc99e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:51.444311 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469675c8-35c2-486e-a932-0403c49cc99e admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:51.444981 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469675c8-35c2-486e-a932-0403c49cc99e admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:51.450809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469675c8-35c2-486e-a932-0403c49cc99e admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:51.450809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469675c8-35c2-486e-a932-0403c49cc99e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:51.450809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469675c8-35c2-486e-a932-0403c49cc99e admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:51.598985 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1120/2238] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:51 2022] POST /identity/v3/projects => generated 373 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:51.602735 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469675c8-35c2-486e-a932-0403c49cc99e admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:51.684369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469675c8-35c2-486e-a932-0403c49cc99e admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:51.688557 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-45c5fdd7-d6a7-49c0-8c8d-80793de01bff admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9ea2c12e28a4df8b4335522cc60b21d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:51.689255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-45c5fdd7-d6a7-49c0-8c8d-80793de01bff admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9ea2c12e28a4df8b4335522cc60b21d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:51.689856 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-45c5fdd7-d6a7-49c0-8c8d-80793de01bff admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9ea2c12e28a4df8b4335522cc60b21d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:51.692925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689a24a0-e5c1-4078-8bae-6e4094efef97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:51.693787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689a24a0-e5c1-4078-8bae-6e4094efef97 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:51.694163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689a24a0-e5c1-4078-8bae-6e4094efef97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:51.695289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689a24a0-e5c1-4078-8bae-6e4094efef97 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:51.696547 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689a24a0-e5c1-4078-8bae-6e4094efef97 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:51.703902 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689a24a0-e5c1-4078-8bae-6e4094efef97 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:51.703996 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-45c5fdd7-d6a7-49c0-8c8d-80793de01bff admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9ea2c12e28a4df8b4335522cc60b21d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:51.745612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:51.767768 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-45c5fdd7-d6a7-49c0-8c8d-80793de01bff admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9ea2c12e28a4df8b4335522cc60b21d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:51.791915 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1119/2239] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:51 2022] DELETE /identity/v3/users/c9ea2c12e28a4df8b4335522cc60b21d => generated 0 bytes in 437 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:51.792959 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c5fdd7-d6a7-49c0-8c8d-80793de01bff admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:51.794580 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-45c5fdd7-d6a7-49c0-8c8d-80793de01bff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:51.821732 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:51.828941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1121/2240] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:51 2022] POST /identity/v3/users => generated 287 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:47:51.833748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689a24a0-e5c1-4078-8bae-6e4094efef97 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:51.835655 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-689a24a0-e5c1-4078-8bae-6e4094efef97 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:51.852306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689a24a0-e5c1-4078-8bae-6e4094efef97 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:51.862381 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decff5e2-d262-4bee-b95e-dc3a26c9892c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:51.863108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decff5e2-d262-4bee-b95e-dc3a26c9892c admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:51.863477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decff5e2-d262-4bee-b95e-dc3a26c9892c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:51.863852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decff5e2-d262-4bee-b95e-dc3a26c9892c admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:51.864599 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decff5e2-d262-4bee-b95e-dc3a26c9892c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:51.866461 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decff5e2-d262-4bee-b95e-dc3a26c9892c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:51.874543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c5fdd7-d6a7-49c0-8c8d-80793de01bff admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:51.880392 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1122/2241] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:51 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:51.884368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decff5e2-d262-4bee-b95e-dc3a26c9892c admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:51.887321 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4931c6a1-4cf2-443f-a553-bd259c02f903 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:51.888017 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4931c6a1-4cf2-443f-a553-bd259c02f903 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:51.888487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4931c6a1-4cf2-443f-a553-bd259c02f903 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:51.888786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4931c6a1-4cf2-443f-a553-bd259c02f903 admin admin] PATH_INFO: `/v3/projects/0860efc0d2584be78c2b46f937ebe7cf` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:51.889920 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4931c6a1-4cf2-443f-a553-bd259c02f903 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0860efc0d2584be78c2b46f937ebe7cf)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:51.891878 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4931c6a1-4cf2-443f-a553-bd259c02f903 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:51.894763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decff5e2-d262-4bee-b95e-dc3a26c9892c admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:51.899889 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa116deb-0c60-43a5-8cb2-3617fdd728e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:51.900826 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa116deb-0c60-43a5-8cb2-3617fdd728e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:51.901106 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa116deb-0c60-43a5-8cb2-3617fdd728e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:51.901441 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa116deb-0c60-43a5-8cb2-3617fdd728e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:51.902245 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa116deb-0c60-43a5-8cb2-3617fdd728e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:51.904050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa116deb-0c60-43a5-8cb2-3617fdd728e5 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:51.917961 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1123/2242] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:51 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:51.923200 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa116deb-0c60-43a5-8cb2-3617fdd728e5 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:51.933510 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa116deb-0c60-43a5-8cb2-3617fdd728e5 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:51.942324 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b011bbc4-f894-4cd1-a9db-5eed663546e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:51.943019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b011bbc4-f894-4cd1-a9db-5eed663546e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:51.946194 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b011bbc4-f894-4cd1-a9db-5eed663546e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:51.946194 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b011bbc4-f894-4cd1-a9db-5eed663546e7 admin admin] PATH_INFO: `/v3/projects/c1eb9ddbc0a84a308ceffeae5198f229/users/587169e32cf64fe5b684e4b2ce0701a8/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:51.961570 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b011bbc4-f894-4cd1-a9db-5eed663546e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c1eb9ddbc0a84a308ceffeae5198f229, user_id=587169e32cf64fe5b684e4b2ce0701a8, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:51.964409 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b011bbc4-f894-4cd1-a9db-5eed663546e7 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:52.080314 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1124/2243] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:51 2022] PUT /identity/v3/projects/c1eb9ddbc0a84a308ceffeae5198f229/users/587169e32cf64fe5b684e4b2ce0701a8/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:52.081311 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b011bbc4-f894-4cd1-a9db-5eed663546e7 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:52.084741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b011bbc4-f894-4cd1-a9db-5eed663546e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:52.097470 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4931c6a1-4cf2-443f-a553-bd259c02f903 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0860efc0d2584be78c2b46f937ebe7cf', 'request_id': 'req-4931c6a1-4cf2-443f-a553-bd259c02f903'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:52.111946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4931c6a1-4cf2-443f-a553-bd259c02f903 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0860efc0d2584be78c2b46f937ebe7cf', 'request_id': 'req-4931c6a1-4cf2-443f-a553-bd259c02f903'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:52.119101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4931c6a1-4cf2-443f-a553-bd259c02f903 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0860efc0d2584be78c2b46f937ebe7cf', 'request_id': 'req-4931c6a1-4cf2-443f-a553-bd259c02f903'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:52.157334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b011bbc4-f894-4cd1-a9db-5eed663546e7 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:52.234819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4931c6a1-4cf2-443f-a553-bd259c02f903 admin admin] The token cache is being invalidate because project 0860efc0d2584be78c2b46f937ebe7cf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:52.234819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4931c6a1-4cf2-443f-a553-bd259c02f903 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:52.234819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4931c6a1-4cf2-443f-a553-bd259c02f903 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:52.234819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1120/2244] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:51 2022] DELETE /identity/v3/projects/0860efc0d2584be78c2b46f937ebe7cf => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:52.235368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c532aba-9ccb-40b6-9fa4-656983c66e74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:52.236673 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c532aba-9ccb-40b6-9fa4-656983c66e74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:52.236673 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c532aba-9ccb-40b6-9fa4-656983c66e74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:52.236673 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c532aba-9ccb-40b6-9fa4-656983c66e74 admin admin] PATH_INFO: `/v3/projects/2829024de36b4ad89d3d1173a7cc3c37` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:52.237309 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8856b119-7deb-4b35-afad-012ebbd4dee6 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:52.237588 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8856b119-7deb-4b35-afad-012ebbd4dee6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:52.237790 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c532aba-9ccb-40b6-9fa4-656983c66e74 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2829024de36b4ad89d3d1173a7cc3c37)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:52.237882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8856b119-7deb-4b35-afad-012ebbd4dee6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:52.239806 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c532aba-9ccb-40b6-9fa4-656983c66e74 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:52.260611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-8856b119-7deb-4b35-afad-012ebbd4dee6 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:52.261883 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8856b119-7deb-4b35-afad-012ebbd4dee6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:52.270391 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1121/2245] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:52.271686 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912441ca-ba21-4906-9ea5-46e19519d59d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:52.271968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912441ca-ba21-4906-9ea5-46e19519d59d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:52.272243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912441ca-ba21-4906-9ea5-46e19519d59d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:52.296255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-912441ca-ba21-4906-9ea5-46e19519d59d None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:52.299854 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-912441ca-ba21-4906-9ea5-46e19519d59d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:52.305706 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1122/2246] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:52.306362 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912441ca-ba21-4906-9ea5-46e19519d59d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:52.307596 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-912441ca-ba21-4906-9ea5-46e19519d59d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:52.357649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=11,keystone:SELECT=87,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:52.370930 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7c532aba-9ccb-40b6-9fa4-656983c66e74 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2829024de36b4ad89d3d1173a7cc3c37', 'request_id': 'req-7c532aba-9ccb-40b6-9fa4-656983c66e74'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:52.376575 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912441ca-ba21-4906-9ea5-46e19519d59d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:52.377373 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7c532aba-9ccb-40b6-9fa4-656983c66e74 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2829024de36b4ad89d3d1173a7cc3c37', 'request_id': 'req-7c532aba-9ccb-40b6-9fa4-656983c66e74'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:52.383438 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7c532aba-9ccb-40b6-9fa4-656983c66e74 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2829024de36b4ad89d3d1173a7cc3c37', 'request_id': 'req-7c532aba-9ccb-40b6-9fa4-656983c66e74'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:52.426803 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6aa4a61-bb77-4201-88c4-992f7e7696e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:52.427409 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6aa4a61-bb77-4201-88c4-992f7e7696e4 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:52.427702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6aa4a61-bb77-4201-88c4-992f7e7696e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:52.427963 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6aa4a61-bb77-4201-88c4-992f7e7696e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:52.485673 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6aa4a61-bb77-4201-88c4-992f7e7696e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:52.487137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6aa4a61-bb77-4201-88c4-992f7e7696e4 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:52.496013 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=93,keystone:INSERT=1,keystone:DELETE=6 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:52.497112 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1123/2247] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:52.498408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acfbb3b-bdef-49d7-a05c-1b6cac4c8156 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:52.498832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acfbb3b-bdef-49d7-a05c-1b6cac4c8156 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:52.500036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acfbb3b-bdef-49d7-a05c-1b6cac4c8156 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:52.502470 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7c532aba-9ccb-40b6-9fa4-656983c66e74 admin admin] The token cache is being invalidate because project 2829024de36b4ad89d3d1173a7cc3c37 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:52.503502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7c532aba-9ccb-40b6-9fa4-656983c66e74 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:52.504343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7c532aba-9ccb-40b6-9fa4-656983c66e74 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:52.506801 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1125/2248] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:52 2022] DELETE /identity/v3/projects/2829024de36b4ad89d3d1173a7cc3c37 => generated 0 bytes in 426 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:52.507920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c532aba-9ccb-40b6-9fa4-656983c66e74 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:52.509452 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7c532aba-9ccb-40b6-9fa4-656983c66e74 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:52.521456 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-6acfbb3b-bdef-49d7-a05c-1b6cac4c8156 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:52.522686 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6acfbb3b-bdef-49d7-a05c-1b6cac4c8156 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:52.523911 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c532aba-9ccb-40b6-9fa4-656983c66e74 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:52.528712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6302f385-6999-4d6e-a2a7-5c90a43c5649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:52.529330 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6302f385-6999-4d6e-a2a7-5c90a43c5649 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:52.529680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6302f385-6999-4d6e-a2a7-5c90a43c5649 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:52.529950 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6302f385-6999-4d6e-a2a7-5c90a43c5649 admin admin] PATH_INFO: `/v3/projects/aba854709eb547f9b7998ff458a00168` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:52.531520 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1124/2249] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:52.536143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e12564-d539-4e03-a938-80e7e1ff3ecb None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:52.536143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e12564-d539-4e03-a938-80e7e1ff3ecb None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:52.536143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e12564-d539-4e03-a938-80e7e1ff3ecb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:52.536419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6302f385-6999-4d6e-a2a7-5c90a43c5649 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aba854709eb547f9b7998ff458a00168)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:52.536419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6302f385-6999-4d6e-a2a7-5c90a43c5649 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:52.553862 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-73e12564-d539-4e03-a938-80e7e1ff3ecb None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:52.555235 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-73e12564-d539-4e03-a938-80e7e1ff3ecb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:52.563403 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1125/2250] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:52.568560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e12564-d539-4e03-a938-80e7e1ff3ecb None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:52.573996 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e12564-d539-4e03-a938-80e7e1ff3ecb None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:52.579044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46072469-289e-4868-899e-612d88b3641f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:52.579634 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46072469-289e-4868-899e-612d88b3641f admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:52.579949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46072469-289e-4868-899e-612d88b3641f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:52.580219 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46072469-289e-4868-899e-612d88b3641f admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:52.580859 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46072469-289e-4868-899e-612d88b3641f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:47:52.581157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46072469-289e-4868-899e-612d88b3641f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:52.582751 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46072469-289e-4868-899e-612d88b3641f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:52.595811 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1126/2251] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:47:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:47:52.597075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f784dcea-158c-46a8-8718-61cfec8a0129 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:52.597518 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f784dcea-158c-46a8-8718-61cfec8a0129 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:52.597807 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f784dcea-158c-46a8-8718-61cfec8a0129 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:52.620310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f784dcea-158c-46a8-8718-61cfec8a0129 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:52.620310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f784dcea-158c-46a8-8718-61cfec8a0129 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:52.626432 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1127/2252] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:52.627041 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f784dcea-158c-46a8-8718-61cfec8a0129 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:52.637447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f784dcea-158c-46a8-8718-61cfec8a0129 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:52.643666 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61b5c0f-7746-4bf0-9c7f-12a62a255489 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:52.644411 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61b5c0f-7746-4bf0-9c7f-12a62a255489 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:52.649175 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61b5c0f-7746-4bf0-9c7f-12a62a255489 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:52.649175 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61b5c0f-7746-4bf0-9c7f-12a62a255489 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:52.649175 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61b5c0f-7746-4bf0-9c7f-12a62a255489 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:52.649175 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61b5c0f-7746-4bf0-9c7f-12a62a255489 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:52.676719 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6302f385-6999-4d6e-a2a7-5c90a43c5649 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aba854709eb547f9b7998ff458a00168', 'request_id': 'req-6302f385-6999-4d6e-a2a7-5c90a43c5649'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:52.683963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6302f385-6999-4d6e-a2a7-5c90a43c5649 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aba854709eb547f9b7998ff458a00168', 'request_id': 'req-6302f385-6999-4d6e-a2a7-5c90a43c5649'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:52.690838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6302f385-6999-4d6e-a2a7-5c90a43c5649 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aba854709eb547f9b7998ff458a00168', 'request_id': 'req-6302f385-6999-4d6e-a2a7-5c90a43c5649'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:52.786376 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6302f385-6999-4d6e-a2a7-5c90a43c5649 admin admin] The token cache is being invalidate because project aba854709eb547f9b7998ff458a00168 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:52.786797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6302f385-6999-4d6e-a2a7-5c90a43c5649 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:52.792529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6302f385-6999-4d6e-a2a7-5c90a43c5649 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:52.792529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1126/2253] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:52 2022] DELETE /identity/v3/projects/aba854709eb547f9b7998ff458a00168 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:52.792529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6302f385-6999-4d6e-a2a7-5c90a43c5649 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:52.792924 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1128/2254] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:52 2022] POST /identity/v3/projects => generated 373 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:52.794352 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cff161-1b41-4a56-ad33-e6a53c2416ba None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:52.794655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cff161-1b41-4a56-ad33-e6a53c2416ba None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:52.794925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cff161-1b41-4a56-ad33-e6a53c2416ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:52.795032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-6302f385-6999-4d6e-a2a7-5c90a43c5649 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:52.818038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-72cff161-1b41-4a56-ad33-e6a53c2416ba None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:52.819285 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-72cff161-1b41-4a56-ad33-e6a53c2416ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:52.870672 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6302f385-6999-4d6e-a2a7-5c90a43c5649 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:52.893146 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1129/2255] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:52.894061 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cff161-1b41-4a56-ad33-e6a53c2416ba None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:52.895553 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-72cff161-1b41-4a56-ad33-e6a53c2416ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:52.908005 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cff161-1b41-4a56-ad33-e6a53c2416ba None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:52.913350 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcaf162-63dd-451b-aa96-6bf80876bcf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:52.914199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcaf162-63dd-451b-aa96-6bf80876bcf7 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:52.914553 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcaf162-63dd-451b-aa96-6bf80876bcf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:52.914945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcaf162-63dd-451b-aa96-6bf80876bcf7 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:52.915644 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcaf162-63dd-451b-aa96-6bf80876bcf7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:52.917867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcaf162-63dd-451b-aa96-6bf80876bcf7 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:52.944004 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db60097b-3096-410c-a727-c25a1d8f5407 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:52.944615 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db60097b-3096-410c-a727-c25a1d8f5407 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:52.944891 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db60097b-3096-410c-a727-c25a1d8f5407 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:52.945162 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db60097b-3096-410c-a727-c25a1d8f5407 admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:52.945801 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db60097b-3096-410c-a727-c25a1d8f5407 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:47:52.946101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db60097b-3096-410c-a727-c25a1d8f5407 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:52.947780 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db60097b-3096-410c-a727-c25a1d8f5407 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:52.959660 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1127/2256] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:47:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:47:52.961045 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b1b831-ece3-47f8-b57f-6d3e68eb503c None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:52.961329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b1b831-ece3-47f8-b57f-6d3e68eb503c None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:52.961603 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b1b831-ece3-47f8-b57f-6d3e68eb503c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:52.981099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-80b1b831-ece3-47f8-b57f-6d3e68eb503c None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:52.982279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-80b1b831-ece3-47f8-b57f-6d3e68eb503c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:52.990679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1128/2257] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:52.991382 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b1b831-ece3-47f8-b57f-6d3e68eb503c None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:53.002753 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b1b831-ece3-47f8-b57f-6d3e68eb503c None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:53.007949 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da98f1b-ce9e-462b-9aa5-039ed8ab2d73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:53.008686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da98f1b-ce9e-462b-9aa5-039ed8ab2d73 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:53.009029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da98f1b-ce9e-462b-9aa5-039ed8ab2d73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:53.009366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da98f1b-ce9e-462b-9aa5-039ed8ab2d73 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:53.010065 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da98f1b-ce9e-462b-9aa5-039ed8ab2d73 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:53.012136 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da98f1b-ce9e-462b-9aa5-039ed8ab2d73 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:53.029689 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1130/2258] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:52 2022] POST /identity/v3/users => generated 287 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:53.034247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcaf162-63dd-451b-aa96-6bf80876bcf7 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:53.044346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcaf162-63dd-451b-aa96-6bf80876bcf7 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:53.049281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ac08cc-71f0-4e11-b11c-e75f6ca98606 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:53.050044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ac08cc-71f0-4e11-b11c-e75f6ca98606 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:53.050331 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ac08cc-71f0-4e11-b11c-e75f6ca98606 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:53.050629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ac08cc-71f0-4e11-b11c-e75f6ca98606 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:53.053079 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ac08cc-71f0-4e11-b11c-e75f6ca98606 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:47:53.053079 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ac08cc-71f0-4e11-b11c-e75f6ca98606 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:53.053397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ac08cc-71f0-4e11-b11c-e75f6ca98606 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:53.068376 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1131/2259] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:47:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:47:53.069216 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ac08cc-71f0-4e11-b11c-e75f6ca98606 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:53.070935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-11ac08cc-71f0-4e11-b11c-e75f6ca98606 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:53.084978 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ac08cc-71f0-4e11-b11c-e75f6ca98606 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:53.090762 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df91db3a-22ca-437e-a1f4-586eb993021b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:53.091387 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df91db3a-22ca-437e-a1f4-586eb993021b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:53.091666 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df91db3a-22ca-437e-a1f4-586eb993021b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:53.091939 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df91db3a-22ca-437e-a1f4-586eb993021b admin admin] PATH_INFO: `/v3/users/08e1631cd74247409d471e0cd640d253` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:53.093119 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df91db3a-22ca-437e-a1f4-586eb993021b admin admin] RBAC: Authorizing `identity:delete_user(user_id=08e1631cd74247409d471e0cd640d253)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:53.095303 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df91db3a-22ca-437e-a1f4-586eb993021b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:53.161185 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1129/2260] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:52 2022] POST /identity/v3/projects => generated 399 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:53.165084 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da98f1b-ce9e-462b-9aa5-039ed8ab2d73 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:53.229775 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da98f1b-ce9e-462b-9aa5-039ed8ab2d73 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:53.234368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9012af8-f748-4010-a2c1-859b6290f126 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:53.235094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9012af8-f748-4010-a2c1-859b6290f126 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:53.235366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9012af8-f748-4010-a2c1-859b6290f126 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:53.235636 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9012af8-f748-4010-a2c1-859b6290f126 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:53.236272 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9012af8-f748-4010-a2c1-859b6290f126 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:53.237856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9012af8-f748-4010-a2c1-859b6290f126 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:53.248389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1130/2261] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:53 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:53.249937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9012af8-f748-4010-a2c1-859b6290f126 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:53.259843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9012af8-f748-4010-a2c1-859b6290f126 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:53.264871 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a5ff61-4aa4-4cd0-8868-c1920fb236b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:53.265633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a5ff61-4aa4-4cd0-8868-c1920fb236b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:53.265915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a5ff61-4aa4-4cd0-8868-c1920fb236b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:53.267576 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a5ff61-4aa4-4cd0-8868-c1920fb236b5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:53.271304 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a5ff61-4aa4-4cd0-8868-c1920fb236b5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:53.271304 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a5ff61-4aa4-4cd0-8868-c1920fb236b5 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:53.307565 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-df91db3a-22ca-437e-a1f4-586eb993021b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '08e1631cd74247409d471e0cd640d253'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:53.308330 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-df91db3a-22ca-437e-a1f4-586eb993021b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '08e1631cd74247409d471e0cd640d253'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:53.308931 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-df91db3a-22ca-437e-a1f4-586eb993021b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '08e1631cd74247409d471e0cd640d253'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:53.317373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-df91db3a-22ca-437e-a1f4-586eb993021b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '08e1631cd74247409d471e0cd640d253'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:53.432380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1131/2262] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:53 2022] POST /identity/v3/projects => generated 359 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:53.432380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a5ff61-4aa4-4cd0-8868-c1920fb236b5 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:53.434646 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-60a5ff61-4aa4-4cd0-8868-c1920fb236b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:53.473941 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:53.474613 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-df91db3a-22ca-437e-a1f4-586eb993021b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '08e1631cd74247409d471e0cd640d253'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:53.493061 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1132/2263] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:53 2022] DELETE /identity/v3/users/08e1631cd74247409d471e0cd640d253 => generated 0 bytes in 424 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:53.494120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df91db3a-22ca-437e-a1f4-586eb993021b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:53.495530 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-df91db3a-22ca-437e-a1f4-586eb993021b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:53.500027 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a5ff61-4aa4-4cd0-8868-c1920fb236b5 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:53.565663 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6553438a-c1c7-419a-b5fa-d51f853e2063 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:53.565663 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6553438a-c1c7-419a-b5fa-d51f853e2063 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:53.565915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6553438a-c1c7-419a-b5fa-d51f853e2063 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:53.566198 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6553438a-c1c7-419a-b5fa-d51f853e2063 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:53.566960 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6553438a-c1c7-419a-b5fa-d51f853e2063 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:53.568981 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6553438a-c1c7-419a-b5fa-d51f853e2063 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:53.575644 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df91db3a-22ca-437e-a1f4-586eb993021b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:53.581114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4955da-729f-4694-9eba-a2519d238d0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:53.581970 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4955da-729f-4694-9eba-a2519d238d0f admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:53.582426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4955da-729f-4694-9eba-a2519d238d0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:53.582907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4955da-729f-4694-9eba-a2519d238d0f admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:53.584841 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4955da-729f-4694-9eba-a2519d238d0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:53.586941 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4955da-729f-4694-9eba-a2519d238d0f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:53.598248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1133/2264] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:53 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:53.599174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4955da-729f-4694-9eba-a2519d238d0f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:53.602880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8c4955da-729f-4694-9eba-a2519d238d0f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:53.615637 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4955da-729f-4694-9eba-a2519d238d0f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:53.620687 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8d67a7-5652-4c50-91f9-de4aa466fa17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:53.621274 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8d67a7-5652-4c50-91f9-de4aa466fa17 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:53.621555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8d67a7-5652-4c50-91f9-de4aa466fa17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:53.621826 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8d67a7-5652-4c50-91f9-de4aa466fa17 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:53.622469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8d67a7-5652-4c50-91f9-de4aa466fa17 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:53.624956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8d67a7-5652-4c50-91f9-de4aa466fa17 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:53.631242 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:53.688916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1132/2265] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:53 2022] POST /identity/v3/users => generated 300 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:53.691831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dbec9d-acda-455b-9681-9776c22486ed None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:53.691831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dbec9d-acda-455b-9681-9776c22486ed None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:53.691831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dbec9d-acda-455b-9681-9776c22486ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:53.709778 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-36dbec9d-acda-455b-9681-9776c22486ed None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:53.711118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-36dbec9d-acda-455b-9681-9776c22486ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:53.719612 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1133/2266] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:53.723600 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dbec9d-acda-455b-9681-9776c22486ed None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:53.733979 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dbec9d-acda-455b-9681-9776c22486ed None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:53.738662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61d3077-f8ed-4961-9057-4c84155d432d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:53.739323 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61d3077-f8ed-4961-9057-4c84155d432d admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:53.739606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61d3077-f8ed-4961-9057-4c84155d432d admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:53.739883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61d3077-f8ed-4961-9057-4c84155d432d admin admin] PATH_INFO: `/v3/projects/828649edae4b4cbaa1176341d5d933b0/users/99ccdcda76aa4458831660357f1f8956/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:53.749238 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1134/2267] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:53 2022] POST /identity/v3/users => generated 280 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:53.750084 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8d67a7-5652-4c50-91f9-de4aa466fa17 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:53.757291 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61d3077-f8ed-4961-9057-4c84155d432d admin admin] RBAC: Authorizing `identity:create_grant(project_id=828649edae4b4cbaa1176341d5d933b0, user_id=99ccdcda76aa4458831660357f1f8956, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:53.759847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8d67a7-5652-4c50-91f9-de4aa466fa17 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:53.760811 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61d3077-f8ed-4961-9057-4c84155d432d admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:53.765978 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3ebc15-2a5a-470c-88c4-ed02a2c1b7ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:53.766576 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3ebc15-2a5a-470c-88c4-ed02a2c1b7ce admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:53.767168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3ebc15-2a5a-470c-88c4-ed02a2c1b7ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:53.767168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3ebc15-2a5a-470c-88c4-ed02a2c1b7ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:53.767784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3ebc15-2a5a-470c-88c4-ed02a2c1b7ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:53.769365 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3ebc15-2a5a-470c-88c4-ed02a2c1b7ce admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:53.780658 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1135/2268] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:53 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:53.781436 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3ebc15-2a5a-470c-88c4-ed02a2c1b7ce admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:53.783091 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1c3ebc15-2a5a-470c-88c4-ed02a2c1b7ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:53.840679 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3ebc15-2a5a-470c-88c4-ed02a2c1b7ce admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:53.842126 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1134/2269] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:53 2022] PUT /identity/v3/projects/828649edae4b4cbaa1176341d5d933b0/users/99ccdcda76aa4458831660357f1f8956/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:53.843542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61d3077-f8ed-4961-9057-4c84155d432d admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:53.934381 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61d3077-f8ed-4961-9057-4c84155d432d admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:53.935678 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597d4d68-7b29-43e6-b3da-bac3147244e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:53.937530 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597d4d68-7b29-43e6-b3da-bac3147244e9 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:53.937850 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597d4d68-7b29-43e6-b3da-bac3147244e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:53.938098 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597d4d68-7b29-43e6-b3da-bac3147244e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:53.942116 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99f6fb3-ac99-496b-9901-ca2b055a28a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:53.942771 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99f6fb3-ac99-496b-9901-ca2b055a28a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:53.943074 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99f6fb3-ac99-496b-9901-ca2b055a28a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:53.943334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99f6fb3-ac99-496b-9901-ca2b055a28a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:53.944131 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99f6fb3-ac99-496b-9901-ca2b055a28a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:53.945749 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99f6fb3-ac99-496b-9901-ca2b055a28a9 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:53.953434 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597d4d68-7b29-43e6-b3da-bac3147244e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:53.955030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597d4d68-7b29-43e6-b3da-bac3147244e9 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:53.960143 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1135/2270] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:53 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:53.961074 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99f6fb3-ac99-496b-9901-ca2b055a28a9 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:53.966792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1136/2271] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:53.968556 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9df4a8c-a961-4db9-a390-5363ee166567 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:53.968956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9df4a8c-a961-4db9-a390-5363ee166567 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:53.969375 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9df4a8c-a961-4db9-a390-5363ee166567 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:53.975045 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99f6fb3-ac99-496b-9901-ca2b055a28a9 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:53.989985 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3161a6-057d-45f7-b29e-d91233a5e0c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:53.990634 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3161a6-057d-45f7-b29e-d91233a5e0c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:53.991076 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3161a6-057d-45f7-b29e-d91233a5e0c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:53.991360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3161a6-057d-45f7-b29e-d91233a5e0c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:53.992027 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3161a6-057d-45f7-b29e-d91233a5e0c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:53.993941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3161a6-057d-45f7-b29e-d91233a5e0c7 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:54.008897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c9df4a8c-a961-4db9-a390-5363ee166567 None None] MFA Rules not processed for user `99ccdcda76aa4458831660357f1f8956`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:54.011115 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c9df4a8c-a961-4db9-a390-5363ee166567 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:54.024097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1136/2272] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:53 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:54.024097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3161a6-057d-45f7-b29e-d91233a5e0c7 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:54.037253 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3161a6-057d-45f7-b29e-d91233a5e0c7 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:54.042759 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95d52d1-7829-4aa0-9832-e3c56a6f07bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:54.043663 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95d52d1-7829-4aa0-9832-e3c56a6f07bd admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:54.043948 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95d52d1-7829-4aa0-9832-e3c56a6f07bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:54.044225 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95d52d1-7829-4aa0-9832-e3c56a6f07bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:54.044880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95d52d1-7829-4aa0-9832-e3c56a6f07bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:54.047615 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95d52d1-7829-4aa0-9832-e3c56a6f07bd admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:54.062271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1137/2273] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:54 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:54.063797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b64a18b-0510-4644-b42d-6d2e7276e5ac None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:54.064090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b64a18b-0510-4644-b42d-6d2e7276e5ac None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:54.064600 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b64a18b-0510-4644-b42d-6d2e7276e5ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:54.103305 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1137/2274] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:53 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:54.105438 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9df4a8c-a961-4db9-a390-5363ee166567 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:54.117689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-5b64a18b-0510-4644-b42d-6d2e7276e5ac None None] MFA Rules not processed for user `587169e32cf64fe5b684e4b2ce0701a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:54.118279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9df4a8c-a961-4db9-a390-5363ee166567 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:54.119362 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-5b64a18b-0510-4644-b42d-6d2e7276e5ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:54.124924 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab494c36-16af-471f-9f0a-f432e78a0e43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:54.126060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab494c36-16af-471f-9f0a-f432e78a0e43 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:54.126565 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab494c36-16af-471f-9f0a-f432e78a0e43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:54.127176 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab494c36-16af-471f-9f0a-f432e78a0e43 admin admin] PATH_INFO: `/v3/projects/d6413896cf7941dc9f287e9e9ef42498/users/cf23c11619e84dec8df2b424bd52ac07/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:54.152714 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab494c36-16af-471f-9f0a-f432e78a0e43 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d6413896cf7941dc9f287e9e9ef42498, user_id=cf23c11619e84dec8df2b424bd52ac07, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:54.190618 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab494c36-16af-471f-9f0a-f432e78a0e43 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:54.191126 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1138/2275] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:54 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:54.191966 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b64a18b-0510-4644-b42d-6d2e7276e5ac None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:54.203891 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b64a18b-0510-4644-b42d-6d2e7276e5ac None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:54.208500 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e750edeb-b425-4bf5-9a31-4b4d73e6e7b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:54.209206 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e750edeb-b425-4bf5-9a31-4b4d73e6e7b4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:54.209491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e750edeb-b425-4bf5-9a31-4b4d73e6e7b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:54.209766 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e750edeb-b425-4bf5-9a31-4b4d73e6e7b4 admin admin] PATH_INFO: `/v3/projects/4714790785e448f7a397a06d0fd36eb2/users/ac27266fe86a43e191535eeefe80249d/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:54.226174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e750edeb-b425-4bf5-9a31-4b4d73e6e7b4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4714790785e448f7a397a06d0fd36eb2, user_id=ac27266fe86a43e191535eeefe80249d, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:54.229077 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e750edeb-b425-4bf5-9a31-4b4d73e6e7b4 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:54.295747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1138/2276] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:54 2022] PUT /identity/v3/projects/d6413896cf7941dc9f287e9e9ef42498/users/cf23c11619e84dec8df2b424bd52ac07/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:54.295747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab494c36-16af-471f-9f0a-f432e78a0e43 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:54.295747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ab494c36-16af-471f-9f0a-f432e78a0e43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:54.368275 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1139/2277] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:54 2022] PUT /identity/v3/projects/4714790785e448f7a397a06d0fd36eb2/users/ac27266fe86a43e191535eeefe80249d/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:54.369370 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e750edeb-b425-4bf5-9a31-4b4d73e6e7b4 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:54.371020 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-e750edeb-b425-4bf5-9a31-4b4d73e6e7b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:54.388093 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab494c36-16af-471f-9f0a-f432e78a0e43 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:54.437619 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e750edeb-b425-4bf5-9a31-4b4d73e6e7b4 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:54.442191 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059c4fcd-2b8b-45ab-84bb-eacd7e12c21c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:54.442822 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059c4fcd-2b8b-45ab-84bb-eacd7e12c21c service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:54.443107 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059c4fcd-2b8b-45ab-84bb-eacd7e12c21c service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:54.447216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059c4fcd-2b8b-45ab-84bb-eacd7e12c21c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:54.455820 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef5ec15-ad19-40f8-9265-42a381294ef9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:54.456596 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef5ec15-ad19-40f8-9265-42a381294ef9 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:54.456978 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef5ec15-ad19-40f8-9265-42a381294ef9 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:54.457420 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef5ec15-ad19-40f8-9265-42a381294ef9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:54.530277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef5ec15-ad19-40f8-9265-42a381294ef9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:54.530940 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059c4fcd-2b8b-45ab-84bb-eacd7e12c21c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:54.532138 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059c4fcd-2b8b-45ab-84bb-eacd7e12c21c service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:54.532350 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef5ec15-ad19-40f8-9265-42a381294ef9 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:54.541925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1139/2278] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:54 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 255 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:54.544121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faea6336-a92e-4963-89d2-883591829e52 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:54.544121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faea6336-a92e-4963-89d2-883591829e52 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:54.544121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faea6336-a92e-4963-89d2-883591829e52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:54.546390 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1140/2279] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:54 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:54.547996 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7818649-78a0-43c4-acd6-646a6c7a5b9d None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:54.548382 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7818649-78a0-43c4-acd6-646a6c7a5b9d None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:54.548657 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7818649-78a0-43c4-acd6-646a6c7a5b9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:54.576605 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-faea6336-a92e-4963-89d2-883591829e52 None None] MFA Rules not processed for user `cf23c11619e84dec8df2b424bd52ac07`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:54.578566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-faea6336-a92e-4963-89d2-883591829e52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:54.590709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-d7818649-78a0-43c4-acd6-646a6c7a5b9d None None] MFA Rules not processed for user `ac27266fe86a43e191535eeefe80249d`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:54.591638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d7818649-78a0-43c4-acd6-646a6c7a5b9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:54.668156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=83,keystone:INSERT=7,keystone:DELETE=10 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:54.679424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1141/2280] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:54 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:54.683713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1140/2281] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:54 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:54.749989 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7818649-78a0-43c4-acd6-646a6c7a5b9d None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:54.752114 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d7818649-78a0-43c4-acd6-646a6c7a5b9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:54.819747 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7818649-78a0-43c4-acd6-646a6c7a5b9d None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:54.824334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a410d8-4d10-4de4-8bff-d9af6f053360 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:54.824958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a410d8-4d10-4de4-8bff-d9af6f053360 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:54.825238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a410d8-4d10-4de4-8bff-d9af6f053360 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:54.825545 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a410d8-4d10-4de4-8bff-d9af6f053360 admin admin] PATH_INFO: `/v3/projects/fa7a5810c8f840f699ba8281407f4a81` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:54.826675 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a410d8-4d10-4de4-8bff-d9af6f053360 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa7a5810c8f840f699ba8281407f4a81)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:54.828622 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a410d8-4d10-4de4-8bff-d9af6f053360 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:54.936067 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-74a410d8-4d10-4de4-8bff-d9af6f053360 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa7a5810c8f840f699ba8281407f4a81', 'request_id': 'req-74a410d8-4d10-4de4-8bff-d9af6f053360'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:54.942632 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-74a410d8-4d10-4de4-8bff-d9af6f053360 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa7a5810c8f840f699ba8281407f4a81', 'request_id': 'req-74a410d8-4d10-4de4-8bff-d9af6f053360'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:54.949669 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-74a410d8-4d10-4de4-8bff-d9af6f053360 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa7a5810c8f840f699ba8281407f4a81', 'request_id': 'req-74a410d8-4d10-4de4-8bff-d9af6f053360'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:55.072252 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-74a410d8-4d10-4de4-8bff-d9af6f053360 admin admin] The token cache is being invalidate because project fa7a5810c8f840f699ba8281407f4a81 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:55.072546 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-74a410d8-4d10-4de4-8bff-d9af6f053360 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:55.073185 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-74a410d8-4d10-4de4-8bff-d9af6f053360 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:55.074950 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1142/2282] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:54 2022] DELETE /identity/v3/projects/fa7a5810c8f840f699ba8281407f4a81 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:55.079908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4422b20-5ec8-4393-990a-a3f7d3f41137 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:55.080192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4422b20-5ec8-4393-990a-a3f7d3f41137 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:55.080667 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4422b20-5ec8-4393-990a-a3f7d3f41137 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:55.100913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b4422b20-5ec8-4393-990a-a3f7d3f41137 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:55.102199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b4422b20-5ec8-4393-990a-a3f7d3f41137 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:55.161107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1141/2283] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:55.169095 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa7037a-7759-40fd-af8f-c610e33ada97 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:55.169380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa7037a-7759-40fd-af8f-c610e33ada97 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:55.169653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa7037a-7759-40fd-af8f-c610e33ada97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:55.174733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:55.199528 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4422b20-5ec8-4393-990a-a3f7d3f41137 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:55.200971 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b4422b20-5ec8-4393-990a-a3f7d3f41137 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:55.222341 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-9fa7037a-7759-40fd-af8f-c610e33ada97 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:55.223628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-9fa7037a-7759-40fd-af8f-c610e33ada97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:55.232759 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1143/2284] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:55.238071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa7037a-7759-40fd-af8f-c610e33ada97 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:55.268512 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa7037a-7759-40fd-af8f-c610e33ada97 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:55.273583 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687d280d-06e1-4b81-9cc7-7eb9185f7b57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:55.274264 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687d280d-06e1-4b81-9cc7-7eb9185f7b57 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:55.274616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687d280d-06e1-4b81-9cc7-7eb9185f7b57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:55.274930 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687d280d-06e1-4b81-9cc7-7eb9185f7b57 admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:55.276509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687d280d-06e1-4b81-9cc7-7eb9185f7b57 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:47:55.276509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687d280d-06e1-4b81-9cc7-7eb9185f7b57 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:55.277625 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687d280d-06e1-4b81-9cc7-7eb9185f7b57 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:55.283793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4422b20-5ec8-4393-990a-a3f7d3f41137 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:55.289015 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6821c209-f80d-4ed9-8f6b-1bda87c0ec7b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:55.289673 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6821c209-f80d-4ed9-8f6b-1bda87c0ec7b service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:55.289948 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6821c209-f80d-4ed9-8f6b-1bda87c0ec7b service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:55.290221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6821c209-f80d-4ed9-8f6b-1bda87c0ec7b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:55.290560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1144/2285] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:47:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:47:55.294910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687d280d-06e1-4b81-9cc7-7eb9185f7b57 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:55.295604 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6821c209-f80d-4ed9-8f6b-1bda87c0ec7b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:55.306161 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687d280d-06e1-4b81-9cc7-7eb9185f7b57 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:55.310959 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f1e450-b024-41c0-a323-bea15dce2b20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:55.311551 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f1e450-b024-41c0-a323-bea15dce2b20 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:55.311829 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f1e450-b024-41c0-a323-bea15dce2b20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:55.312105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f1e450-b024-41c0-a323-bea15dce2b20 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:55.312758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f1e450-b024-41c0-a323-bea15dce2b20 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:55.314697 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f1e450-b024-41c0-a323-bea15dce2b20 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:55.371963 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6821c209-f80d-4ed9-8f6b-1bda87c0ec7b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:55.373402 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6821c209-f80d-4ed9-8f6b-1bda87c0ec7b service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:55.383337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1142/2286] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:55 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:55.391783 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6821c209-f80d-4ed9-8f6b-1bda87c0ec7b service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:55.391783 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6821c209-f80d-4ed9-8f6b-1bda87c0ec7b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:55.404055 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6821c209-f80d-4ed9-8f6b-1bda87c0ec7b service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:55.410164 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7636f152-15b3-47b2-a32e-b15cbabe6c2e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:55.411271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7636f152-15b3-47b2-a32e-b15cbabe6c2e service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:55.411695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7636f152-15b3-47b2-a32e-b15cbabe6c2e service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:55.412121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7636f152-15b3-47b2-a32e-b15cbabe6c2e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:55.414251 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7636f152-15b3-47b2-a32e-b15cbabe6c2e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:55.422541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1145/2287] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:55 2022] POST /identity/v3/projects => generated 383 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:55.426497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f1e450-b024-41c0-a323-bea15dce2b20 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:55.494176 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7636f152-15b3-47b2-a32e-b15cbabe6c2e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:55.496655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7636f152-15b3-47b2-a32e-b15cbabe6c2e service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:55.509371 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1143/2288] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:55 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:55.533420 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7636f152-15b3-47b2-a32e-b15cbabe6c2e service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:55.535612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7636f152-15b3-47b2-a32e-b15cbabe6c2e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:55.549667 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7636f152-15b3-47b2-a32e-b15cbabe6c2e service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:55.558695 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f1e450-b024-41c0-a323-bea15dce2b20 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:55.564683 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6361b7-693c-422b-b022-2faca5fec643 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:55.565544 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6361b7-693c-422b-b022-2faca5fec643 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:55.565544 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6361b7-693c-422b-b022-2faca5fec643 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:55.565952 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6361b7-693c-422b-b022-2faca5fec643 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:55.566832 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6361b7-693c-422b-b022-2faca5fec643 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:55.568892 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6361b7-693c-422b-b022-2faca5fec643 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:55.570676 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a376cfe-4394-4f04-b539-8da115bb8206 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:55.571446 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a376cfe-4394-4f04-b539-8da115bb8206 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:55.583056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a376cfe-4394-4f04-b539-8da115bb8206 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:55.583056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a376cfe-4394-4f04-b539-8da115bb8206 admin admin] PATH_INFO: `/v3/users/ac27266fe86a43e191535eeefe80249d` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:55.583919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a376cfe-4394-4f04-b539-8da115bb8206 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ac27266fe86a43e191535eeefe80249d)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:55.585881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a376cfe-4394-4f04-b539-8da115bb8206 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:55.672051 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1146/2289] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:55 2022] POST /identity/v3/users => generated 292 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:47:55.709102 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6361b7-693c-422b-b022-2faca5fec643 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:55.719918 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6361b7-693c-422b-b022-2faca5fec643 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:55.728527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89d5fc8-c344-44e9-8f00-d2679ec8bd82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:55.729231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89d5fc8-c344-44e9-8f00-d2679ec8bd82 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:55.729507 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89d5fc8-c344-44e9-8f00-d2679ec8bd82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:55.730087 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89d5fc8-c344-44e9-8f00-d2679ec8bd82 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:55.731570 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89d5fc8-c344-44e9-8f00-d2679ec8bd82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:55.732208 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89d5fc8-c344-44e9-8f00-d2679ec8bd82 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:55.742370 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1147/2290] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:55 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:47:55.746710 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89d5fc8-c344-44e9-8f00-d2679ec8bd82 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:55.756574 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89d5fc8-c344-44e9-8f00-d2679ec8bd82 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:55.761178 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0403e9b4-2a78-4b3d-a9ae-027968e81dac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:55.763311 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0403e9b4-2a78-4b3d-a9ae-027968e81dac admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:55.763311 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0403e9b4-2a78-4b3d-a9ae-027968e81dac admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:55.763311 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0403e9b4-2a78-4b3d-a9ae-027968e81dac admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:55.763311 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0403e9b4-2a78-4b3d-a9ae-027968e81dac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:55.765072 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0403e9b4-2a78-4b3d-a9ae-027968e81dac admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:55.775757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1148/2291] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:55 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:47:55.792066 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0403e9b4-2a78-4b3d-a9ae-027968e81dac admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:55.792066 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0403e9b4-2a78-4b3d-a9ae-027968e81dac admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:55.797061 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf8d563-e776-496d-a781-08f6ca6c04a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:55.797697 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf8d563-e776-496d-a781-08f6ca6c04a0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:55.797974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf8d563-e776-496d-a781-08f6ca6c04a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:55.798242 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf8d563-e776-496d-a781-08f6ca6c04a0 admin admin] PATH_INFO: `/v3/projects/0be64f8f1d4b452e84a7bcd64f6100d1/users/4d46ec34caf74ea7b7c8f836c2f7ea6c/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:55.811207 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:55.815841 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf8d563-e776-496d-a781-08f6ca6c04a0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0be64f8f1d4b452e84a7bcd64f6100d1, user_id=4d46ec34caf74ea7b7c8f836c2f7ea6c, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:55.818700 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf8d563-e776-496d-a781-08f6ca6c04a0 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:55.824962 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8a376cfe-4394-4f04-b539-8da115bb8206 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ac27266fe86a43e191535eeefe80249d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:55.825709 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8a376cfe-4394-4f04-b539-8da115bb8206 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ac27266fe86a43e191535eeefe80249d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:55.826337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8a376cfe-4394-4f04-b539-8da115bb8206 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ac27266fe86a43e191535eeefe80249d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:55.834786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8a376cfe-4394-4f04-b539-8da115bb8206 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ac27266fe86a43e191535eeefe80249d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:55.911573 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8a376cfe-4394-4f04-b539-8da115bb8206 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ac27266fe86a43e191535eeefe80249d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:55.917264 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1149/2292] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:55 2022] PUT /identity/v3/projects/0be64f8f1d4b452e84a7bcd64f6100d1/users/4d46ec34caf74ea7b7c8f836c2f7ea6c/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:55.922303 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a1fe72-b950-46a8-90dd-3c5b0f732004 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:55.922600 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a1fe72-b950-46a8-90dd-3c5b0f732004 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:55.922897 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a1fe72-b950-46a8-90dd-3c5b0f732004 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:55.930148 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1144/2293] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:55 2022] DELETE /identity/v3/users/ac27266fe86a43e191535eeefe80249d => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:55.934365 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d75a8e-24ed-454f-8213-8d54491cd777 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:55.934710 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d75a8e-24ed-454f-8213-8d54491cd777 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:55.934995 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d75a8e-24ed-454f-8213-8d54491cd777 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:55.942304 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-16a1fe72-b950-46a8-90dd-3c5b0f732004 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:55.946933 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-16a1fe72-b950-46a8-90dd-3c5b0f732004 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:55.954767 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-95d75a8e-24ed-454f-8213-8d54491cd777 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:55.955954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-95d75a8e-24ed-454f-8213-8d54491cd777 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:56.004496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1150/2294] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:47:56.014983 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a1fe72-b950-46a8-90dd-3c5b0f732004 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:56.018326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:56.019732 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-16a1fe72-b950-46a8-90dd-3c5b0f732004 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:56.022913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1145/2295] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:47:56.040559 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d75a8e-24ed-454f-8213-8d54491cd777 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:56.041941 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-95d75a8e-24ed-454f-8213-8d54491cd777 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:56.082492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a1fe72-b950-46a8-90dd-3c5b0f732004 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:56.090342 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e06072-5382-40dc-90bf-c7322d50614c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:56.090970 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e06072-5382-40dc-90bf-c7322d50614c service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:56.091254 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e06072-5382-40dc-90bf-c7322d50614c service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:56.091527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e06072-5382-40dc-90bf-c7322d50614c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:56.100827 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e06072-5382-40dc-90bf-c7322d50614c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:56.102273 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e06072-5382-40dc-90bf-c7322d50614c service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:56.110114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d75a8e-24ed-454f-8213-8d54491cd777 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:56.113084 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1151/2296] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:47:56.114070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecc401a-a852-42e1-b881-0d05c080324b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:56.114732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecc401a-a852-42e1-b881-0d05c080324b service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:56.115013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecc401a-a852-42e1-b881-0d05c080324b service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:56.115283 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecc401a-a852-42e1-b881-0d05c080324b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:56.125517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eecc401a-a852-42e1-b881-0d05c080324b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:56.126965 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eecc401a-a852-42e1-b881-0d05c080324b service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:56.136522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1146/2297] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:47:56.639748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e06072-5382-40dc-90bf-c7322d50614c service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:56.641369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-07e06072-5382-40dc-90bf-c7322d50614c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:56.666752 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e06072-5382-40dc-90bf-c7322d50614c service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:56.679192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ea4229-866d-486b-b533-99297a0e477a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:56.680832 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ea4229-866d-486b-b533-99297a0e477a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:56.681211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ea4229-866d-486b-b533-99297a0e477a admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:56.681768 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ea4229-866d-486b-b533-99297a0e477a admin admin] PATH_INFO: `/v3/projects/4714790785e448f7a397a06d0fd36eb2` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:56.688449 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ea4229-866d-486b-b533-99297a0e477a admin admin] RBAC: Authorizing `identity:delete_project(project_id=4714790785e448f7a397a06d0fd36eb2)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:56.688449 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ea4229-866d-486b-b533-99297a0e477a admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:56.828770 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:56.851634 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-47ea4229-866d-486b-b533-99297a0e477a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4714790785e448f7a397a06d0fd36eb2', 'request_id': 'req-47ea4229-866d-486b-b533-99297a0e477a'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:56.859796 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-47ea4229-866d-486b-b533-99297a0e477a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4714790785e448f7a397a06d0fd36eb2', 'request_id': 'req-47ea4229-866d-486b-b533-99297a0e477a'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:56.871062 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-47ea4229-866d-486b-b533-99297a0e477a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4714790785e448f7a397a06d0fd36eb2', 'request_id': 'req-47ea4229-866d-486b-b533-99297a0e477a'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:56.950507 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-47ea4229-866d-486b-b533-99297a0e477a admin admin] The token cache is being invalidate because project 4714790785e448f7a397a06d0fd36eb2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:47:56.950507 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-47ea4229-866d-486b-b533-99297a0e477a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:56.950507 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-47ea4229-866d-486b-b533-99297a0e477a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:56.951933 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1152/2298] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:56 2022] DELETE /identity/v3/projects/4714790785e448f7a397a06d0fd36eb2 => generated 0 bytes in 313 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:56.957018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a75f2e-da1b-46dc-9278-9e704b68843b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:56.957423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a75f2e-da1b-46dc-9278-9e704b68843b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:56.957951 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a75f2e-da1b-46dc-9278-9e704b68843b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:56.982818 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-71a75f2e-da1b-46dc-9278-9e704b68843b None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:56.984758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-71a75f2e-da1b-46dc-9278-9e704b68843b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:57.049992 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1147/2299] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:57.057629 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db00979-4a62-450f-b772-dd239db94c83 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:57.057919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db00979-4a62-450f-b772-dd239db94c83 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:57.058196 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db00979-4a62-450f-b772-dd239db94c83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:57.078538 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-4db00979-4a62-450f-b772-dd239db94c83 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:57.079778 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-4db00979-4a62-450f-b772-dd239db94c83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:57.087892 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1153/2300] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:47:57.092409 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a75f2e-da1b-46dc-9278-9e704b68843b None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:57.104577 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a75f2e-da1b-46dc-9278-9e704b68843b None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:57.109550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0c71f7-e40d-473c-881f-c0174955ac5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:57.110412 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0c71f7-e40d-473c-881f-c0174955ac5e admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:57.110688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0c71f7-e40d-473c-881f-c0174955ac5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:57.111013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0c71f7-e40d-473c-881f-c0174955ac5e admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:57.111673 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0c71f7-e40d-473c-881f-c0174955ac5e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:47:57.111994 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0c71f7-e40d-473c-881f-c0174955ac5e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:57.117091 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0c71f7-e40d-473c-881f-c0174955ac5e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:57.131041 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:57.131142 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1148/2301] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:47:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:47:57.132132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db00979-4a62-450f-b772-dd239db94c83 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:57.141573 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db00979-4a62-450f-b772-dd239db94c83 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:57.146301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcbfb62b-5f84-49a6-9ba4-fdcda397642a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:57.146952 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbfb62b-5f84-49a6-9ba4-fdcda397642a admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:57.147204 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbfb62b-5f84-49a6-9ba4-fdcda397642a admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:57.147567 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbfb62b-5f84-49a6-9ba4-fdcda397642a admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:57.148099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcbfb62b-5f84-49a6-9ba4-fdcda397642a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:57.150012 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcbfb62b-5f84-49a6-9ba4-fdcda397642a admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:57.257033 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1154/2302] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:47:57 2022] POST /identity/v3/projects => generated 375 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:57.261286 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0c71f7-e40d-473c-881f-c0174955ac5e admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:57.272911 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0c71f7-e40d-473c-881f-c0174955ac5e admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:57.332019 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:57.338998 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a33a2d-d941-493c-95a6-5694b5685629 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:57.339795 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a33a2d-d941-493c-95a6-5694b5685629 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:57.340380 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a33a2d-d941-493c-95a6-5694b5685629 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:57.340661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a33a2d-d941-493c-95a6-5694b5685629 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:57.341411 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a33a2d-d941-493c-95a6-5694b5685629 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:57.343912 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a33a2d-d941-493c-95a6-5694b5685629 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:57.437429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1149/2303] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:47:57 2022] POST /identity/v3/users => generated 288 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:47:57.437550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcbfb62b-5f84-49a6-9ba4-fdcda397642a admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:57.448252 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcbfb62b-5f84-49a6-9ba4-fdcda397642a admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:57.453555 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac218d2-3032-4203-b307-90bb369cb25c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:57.454474 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac218d2-3032-4203-b307-90bb369cb25c admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:57.455361 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac218d2-3032-4203-b307-90bb369cb25c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:57.455653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac218d2-3032-4203-b307-90bb369cb25c admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:57.456356 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac218d2-3032-4203-b307-90bb369cb25c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:57.458003 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac218d2-3032-4203-b307-90bb369cb25c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:57.469327 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d245f0-fe99-4a60-aadc-2d9cd4ecc02b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:57.477830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d245f0-fe99-4a60-aadc-2d9cd4ecc02b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:57.477830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d245f0-fe99-4a60-aadc-2d9cd4ecc02b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:57.484272 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1155/2304] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:57 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:57.485396 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac218d2-3032-4203-b307-90bb369cb25c admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:57.490719 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d5d245f0-fe99-4a60-aadc-2d9cd4ecc02b None None] MFA Rules not processed for user `587169e32cf64fe5b684e4b2ce0701a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:57.492326 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d5d245f0-fe99-4a60-aadc-2d9cd4ecc02b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:57.496735 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac218d2-3032-4203-b307-90bb369cb25c admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:57.502885 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4330bd36-835f-450b-ba75-4ce1c1f713d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:57.503668 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4330bd36-835f-450b-ba75-4ce1c1f713d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:57.514733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4330bd36-835f-450b-ba75-4ce1c1f713d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:57.514733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4330bd36-835f-450b-ba75-4ce1c1f713d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:57.514733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4330bd36-835f-450b-ba75-4ce1c1f713d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:57.514733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4330bd36-835f-450b-ba75-4ce1c1f713d3 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:57.517968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1156/2305] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:47:57 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:47:57.521977 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4330bd36-835f-450b-ba75-4ce1c1f713d3 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:57.532077 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4330bd36-835f-450b-ba75-4ce1c1f713d3 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:57.538449 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba22512-3213-4334-a17c-f5c8dc602742 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:57.539396 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba22512-3213-4334-a17c-f5c8dc602742 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:57.539509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba22512-3213-4334-a17c-f5c8dc602742 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:57.539848 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba22512-3213-4334-a17c-f5c8dc602742 admin admin] PATH_INFO: `/v3/projects/3aced3101122427ebab228330526aaca/users/135fe9bb598d4a489ceea853be586c03/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:57.571766 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba22512-3213-4334-a17c-f5c8dc602742 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3aced3101122427ebab228330526aaca, user_id=135fe9bb598d4a489ceea853be586c03, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:57.576696 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba22512-3213-4334-a17c-f5c8dc602742 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:57.578129 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1150/2306] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:57 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:47:57.583797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:57.668766 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1157/2307] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:47:57 2022] PUT /identity/v3/projects/3aced3101122427ebab228330526aaca/users/135fe9bb598d4a489ceea853be586c03/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:47:57.680895 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e684a3b-e0bc-480e-b084-e6aff888d3a9 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:57.681200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e684a3b-e0bc-480e-b084-e6aff888d3a9 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:57.681455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e684a3b-e0bc-480e-b084-e6aff888d3a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:57.715773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-1e684a3b-e0bc-480e-b084-e6aff888d3a9 None None] MFA Rules not processed for user `135fe9bb598d4a489ceea853be586c03`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:57.717848 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1e684a3b-e0bc-480e-b084-e6aff888d3a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:57.809782 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1151/2308] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:57 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:47:57.821678 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba22512-3213-4334-a17c-f5c8dc602742 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:57.823607 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-5ba22512-3213-4334-a17c-f5c8dc602742 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:57.828619 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:57.908194 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba22512-3213-4334-a17c-f5c8dc602742 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:57.912533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9611c37b-6584-4c2f-bc8c-61a21a46feba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:57.913265 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9611c37b-6584-4c2f-bc8c-61a21a46feba service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:57.913651 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9611c37b-6584-4c2f-bc8c-61a21a46feba service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:57.914048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9611c37b-6584-4c2f-bc8c-61a21a46feba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:57.924111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9611c37b-6584-4c2f-bc8c-61a21a46feba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:57.925916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9611c37b-6584-4c2f-bc8c-61a21a46feba service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:57.936309 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1158/2309] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:57 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:58.284173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e684a3b-e0bc-480e-b084-e6aff888d3a9 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:58.285899 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1e684a3b-e0bc-480e-b084-e6aff888d3a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:58.433623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e684a3b-e0bc-480e-b084-e6aff888d3a9 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:58.445137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cc5859-2195-4bdf-a954-a98a38f5636e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:58.446242 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cc5859-2195-4bdf-a954-a98a38f5636e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:58.447294 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cc5859-2195-4bdf-a954-a98a38f5636e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:58.447937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cc5859-2195-4bdf-a954-a98a38f5636e admin admin] PATH_INFO: `/v3/users/99ccdcda76aa4458831660357f1f8956` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:58.450426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43cc5859-2195-4bdf-a954-a98a38f5636e admin admin] RBAC: Authorizing `identity:delete_user(user_id=99ccdcda76aa4458831660357f1f8956)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:58.454577 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43cc5859-2195-4bdf-a954-a98a38f5636e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:58.749308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-43cc5859-2195-4bdf-a954-a98a38f5636e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '99ccdcda76aa4458831660357f1f8956'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:58.750431 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-43cc5859-2195-4bdf-a954-a98a38f5636e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '99ccdcda76aa4458831660357f1f8956'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:58.751855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-43cc5859-2195-4bdf-a954-a98a38f5636e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '99ccdcda76aa4458831660357f1f8956'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:58.762739 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-43cc5859-2195-4bdf-a954-a98a38f5636e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '99ccdcda76aa4458831660357f1f8956'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:58.901988 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-43cc5859-2195-4bdf-a954-a98a38f5636e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '99ccdcda76aa4458831660357f1f8956'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:58.944469 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1152/2310] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:47:58 2022] DELETE /identity/v3/users/99ccdcda76aa4458831660357f1f8956 => generated 0 bytes in 661 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:47:58.948818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e28d5c8-559f-41dc-bbc1-8f3b74069d70 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:58.949375 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e28d5c8-559f-41dc-bbc1-8f3b74069d70 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:58.949476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e28d5c8-559f-41dc-bbc1-8f3b74069d70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:58.975744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-3e28d5c8-559f-41dc-bbc1-8f3b74069d70 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:47:58.977267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-3e28d5c8-559f-41dc-bbc1-8f3b74069d70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:59.030539 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:59.055726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1159/2311] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:47:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:47:59.066196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cc5859-2195-4bdf-a954-a98a38f5636e admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:59.067494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-43cc5859-2195-4bdf-a954-a98a38f5636e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:59.133343 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cc5859-2195-4bdf-a954-a98a38f5636e admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:59.137363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8e0eb3-c10d-454d-90f3-1b7c477cab1d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:59.140685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8e0eb3-c10d-454d-90f3-1b7c477cab1d service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:59.140685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8e0eb3-c10d-454d-90f3-1b7c477cab1d service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:59.140685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8e0eb3-c10d-454d-90f3-1b7c477cab1d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:59.151691 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8e0eb3-c10d-454d-90f3-1b7c477cab1d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:59.153196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8e0eb3-c10d-454d-90f3-1b7c477cab1d service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:59.163880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1153/2312] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:47:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:47:59.359196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:47:59.730347 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e28d5c8-559f-41dc-bbc1-8f3b74069d70 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:47:59.732605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-3e28d5c8-559f-41dc-bbc1-8f3b74069d70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:47:59.769965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e28d5c8-559f-41dc-bbc1-8f3b74069d70 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:47:59.775859 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ac0440-ecdb-48d6-9022-a262d4cf6a26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:47:59.776768 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ac0440-ecdb-48d6-9022-a262d4cf6a26 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:47:59.777258 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ac0440-ecdb-48d6-9022-a262d4cf6a26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:47:59.777716 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ac0440-ecdb-48d6-9022-a262d4cf6a26 admin admin] PATH_INFO: `/v3/projects/828649edae4b4cbaa1176341d5d933b0` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:47:59.779204 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ac0440-ecdb-48d6-9022-a262d4cf6a26 admin admin] RBAC: Authorizing `identity:delete_project(project_id=828649edae4b4cbaa1176341d5d933b0)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:47:59.781560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ac0440-ecdb-48d6-9022-a262d4cf6a26 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:47:59.897411 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-b9ac0440-ecdb-48d6-9022-a262d4cf6a26 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '828649edae4b4cbaa1176341d5d933b0', 'request_id': 'req-b9ac0440-ecdb-48d6-9022-a262d4cf6a26'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:59.904360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-b9ac0440-ecdb-48d6-9022-a262d4cf6a26 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '828649edae4b4cbaa1176341d5d933b0', 'request_id': 'req-b9ac0440-ecdb-48d6-9022-a262d4cf6a26'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:47:59.916227 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-b9ac0440-ecdb-48d6-9022-a262d4cf6a26 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '828649edae4b4cbaa1176341d5d933b0', 'request_id': 'req-b9ac0440-ecdb-48d6-9022-a262d4cf6a26'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:00.006495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-b9ac0440-ecdb-48d6-9022-a262d4cf6a26 admin admin] The token cache is being invalidate because project 828649edae4b4cbaa1176341d5d933b0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:48:00.007220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-b9ac0440-ecdb-48d6-9022-a262d4cf6a26 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:00.007540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-b9ac0440-ecdb-48d6-9022-a262d4cf6a26 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:00.009104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1160/2313] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:47:59 2022] DELETE /identity/v3/projects/828649edae4b4cbaa1176341d5d933b0 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:00.014738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a0e60f-beb5-42be-99d2-802162839cc2 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:00.015099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a0e60f-beb5-42be-99d2-802162839cc2 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:00.015449 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a0e60f-beb5-42be-99d2-802162839cc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:00.044499 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c2a0e60f-beb5-42be-99d2-802162839cc2 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:00.046796 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c2a0e60f-beb5-42be-99d2-802162839cc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:00.134937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1154/2314] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:00.144070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32131c43-c31f-489c-a0b3-4d6b246be675 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:00.144355 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32131c43-c31f-489c-a0b3-4d6b246be675 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:00.144634 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32131c43-c31f-489c-a0b3-4d6b246be675 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:00.170686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-32131c43-c31f-489c-a0b3-4d6b246be675 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:00.170686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-32131c43-c31f-489c-a0b3-4d6b246be675 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:00.175166 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1161/2315] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:00.184153 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a0e60f-beb5-42be-99d2-802162839cc2 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:00.195956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a0e60f-beb5-42be-99d2-802162839cc2 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:00.205069 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e625d3fb-deff-4833-a38c-4b1c110156de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:00.205942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e625d3fb-deff-4833-a38c-4b1c110156de admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:00.206394 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e625d3fb-deff-4833-a38c-4b1c110156de admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:00.206865 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e625d3fb-deff-4833-a38c-4b1c110156de admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:00.207698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e625d3fb-deff-4833-a38c-4b1c110156de admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:48:00.208161 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e625d3fb-deff-4833-a38c-4b1c110156de admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:00.210130 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e625d3fb-deff-4833-a38c-4b1c110156de admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:00.225836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1155/2316] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:48:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:48:00.244147 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32131c43-c31f-489c-a0b3-4d6b246be675 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:00.253877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32131c43-c31f-489c-a0b3-4d6b246be675 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:00.259009 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beb15df-63da-466e-9db7-69182a509306 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:00.259613 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beb15df-63da-466e-9db7-69182a509306 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:00.259893 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beb15df-63da-466e-9db7-69182a509306 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:00.260165 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beb15df-63da-466e-9db7-69182a509306 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:00.260795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6beb15df-63da-466e-9db7-69182a509306 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:00.262859 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6beb15df-63da-466e-9db7-69182a509306 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:00.441216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1162/2317] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:48:00 2022] POST /identity/v3/projects => generated 373 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:48:00.445674 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e625d3fb-deff-4833-a38c-4b1c110156de admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:00.456541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e625d3fb-deff-4833-a38c-4b1c110156de admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:00.526240 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4363b7e6-1cc1-470b-ac61-0a3282398171 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:00.526991 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4363b7e6-1cc1-470b-ac61-0a3282398171 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:00.527342 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4363b7e6-1cc1-470b-ac61-0a3282398171 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:00.527614 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4363b7e6-1cc1-470b-ac61-0a3282398171 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:00.529518 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4363b7e6-1cc1-470b-ac61-0a3282398171 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:00.530647 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4363b7e6-1cc1-470b-ac61-0a3282398171 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:00.658549 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1156/2318] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:48:00 2022] POST /identity/v3/users => generated 287 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:48:00.662572 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beb15df-63da-466e-9db7-69182a509306 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:00.672976 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:00.676359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beb15df-63da-466e-9db7-69182a509306 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:00.683315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069a33aa-55af-421f-a23b-0aab10c29eea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:00.684003 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069a33aa-55af-421f-a23b-0aab10c29eea admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:00.684540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069a33aa-55af-421f-a23b-0aab10c29eea admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:00.684872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069a33aa-55af-421f-a23b-0aab10c29eea admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:00.686236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069a33aa-55af-421f-a23b-0aab10c29eea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:00.688100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069a33aa-55af-421f-a23b-0aab10c29eea admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:00.699064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1163/2319] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:00 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:48:00.704008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4363b7e6-1cc1-470b-ac61-0a3282398171 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:00.714847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4363b7e6-1cc1-470b-ac61-0a3282398171 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:00.720090 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38850550-6b84-4019-a1a2-09e167775a86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:00.720709 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38850550-6b84-4019-a1a2-09e167775a86 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:00.720984 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38850550-6b84-4019-a1a2-09e167775a86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:00.721311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38850550-6b84-4019-a1a2-09e167775a86 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:00.722094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38850550-6b84-4019-a1a2-09e167775a86 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:00.723749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38850550-6b84-4019-a1a2-09e167775a86 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:00.741679 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1157/2320] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:00 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:48:00.743027 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069a33aa-55af-421f-a23b-0aab10c29eea admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:00.755837 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069a33aa-55af-421f-a23b-0aab10c29eea admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:00.761142 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8295ab14-c8e7-4540-a84b-ae2f951f63c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:00.761813 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8295ab14-c8e7-4540-a84b-ae2f951f63c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:00.762232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8295ab14-c8e7-4540-a84b-ae2f951f63c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:00.762513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8295ab14-c8e7-4540-a84b-ae2f951f63c2 admin admin] PATH_INFO: `/v3/projects/b9df2c8224e94ba994c3971cf81dca4e/users/d4c198a8faba41c08ef15593a6f39642/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:00.781875 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8295ab14-c8e7-4540-a84b-ae2f951f63c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b9df2c8224e94ba994c3971cf81dca4e, user_id=d4c198a8faba41c08ef15593a6f39642, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:00.784608 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8295ab14-c8e7-4540-a84b-ae2f951f63c2 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:00.858487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1164/2321] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:48:00 2022] PUT /identity/v3/projects/b9df2c8224e94ba994c3971cf81dca4e/users/d4c198a8faba41c08ef15593a6f39642/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:00.869792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c1af7f-2e4c-493d-9eed-ca4442062256 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:00.870177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c1af7f-2e4c-493d-9eed-ca4442062256 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:00.870747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c1af7f-2e4c-493d-9eed-ca4442062256 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:00.872881 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=96,keystone:INSERT=2,keystone:DELETE=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:00.908013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c9c1af7f-2e4c-493d-9eed-ca4442062256 None None] MFA Rules not processed for user `d4c198a8faba41c08ef15593a6f39642`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:00.909338 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c9c1af7f-2e4c-493d-9eed-ca4442062256 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:01.007017 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1158/2322] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:00 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:01.011693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8295ab14-c8e7-4540-a84b-ae2f951f63c2 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:01.096081 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8295ab14-c8e7-4540-a84b-ae2f951f63c2 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:01.101162 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae78f26f-4fa9-496e-96f7-b9d9e661b230 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:01.115435 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae78f26f-4fa9-496e-96f7-b9d9e661b230 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:01.115435 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae78f26f-4fa9-496e-96f7-b9d9e661b230 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:01.115435 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae78f26f-4fa9-496e-96f7-b9d9e661b230 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:01.115435 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae78f26f-4fa9-496e-96f7-b9d9e661b230 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:01.115435 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae78f26f-4fa9-496e-96f7-b9d9e661b230 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:01.252506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1165/2323] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:48:01 2022] POST /identity/v3/projects => generated 371 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:48:01.253982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c1af7f-2e4c-493d-9eed-ca4442062256 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:01.266153 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c1af7f-2e4c-493d-9eed-ca4442062256 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:01.386159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35741fe4-9502-4a76-b395-c576fe42d055 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:01.387307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35741fe4-9502-4a76-b395-c576fe42d055 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:01.387307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35741fe4-9502-4a76-b395-c576fe42d055 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:01.387556 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35741fe4-9502-4a76-b395-c576fe42d055 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:01.388186 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35741fe4-9502-4a76-b395-c576fe42d055 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:01.390297 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35741fe4-9502-4a76-b395-c576fe42d055 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:01.487779 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1159/2324] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:48:01 2022] POST /identity/v3/users => generated 292 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:48:01.488488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae78f26f-4fa9-496e-96f7-b9d9e661b230 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:01.503924 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae78f26f-4fa9-496e-96f7-b9d9e661b230 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:01.510226 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24d6eea-fe7d-4276-9652-55af9b278c14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:01.511557 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24d6eea-fe7d-4276-9652-55af9b278c14 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:01.512081 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24d6eea-fe7d-4276-9652-55af9b278c14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:01.512627 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24d6eea-fe7d-4276-9652-55af9b278c14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:01.513493 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24d6eea-fe7d-4276-9652-55af9b278c14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:01.533279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24d6eea-fe7d-4276-9652-55af9b278c14 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:01.555662 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35741fe4-9502-4a76-b395-c576fe42d055 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:01.555859 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1166/2325] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:01 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:48:01.570672 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35741fe4-9502-4a76-b395-c576fe42d055 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:01.579879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04583e63-2ca1-4975-86bc-5d75bec5f857 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:01.580855 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04583e63-2ca1-4975-86bc-5d75bec5f857 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:01.581318 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04583e63-2ca1-4975-86bc-5d75bec5f857 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:01.582306 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04583e63-2ca1-4975-86bc-5d75bec5f857 admin admin] PATH_INFO: `/v3/projects/ccfb4ade5d154d08a2df76f8b19cb8bf/users/c61a1c1280b6421ea4ad99e38538b09b/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:01.613966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04583e63-2ca1-4975-86bc-5d75bec5f857 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ccfb4ade5d154d08a2df76f8b19cb8bf, user_id=c61a1c1280b6421ea4ad99e38538b09b, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:01.617702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04583e63-2ca1-4975-86bc-5d75bec5f857 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:01.751323 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1160/2326] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:48:01 2022] PUT /identity/v3/projects/ccfb4ade5d154d08a2df76f8b19cb8bf/users/c61a1c1280b6421ea4ad99e38538b09b/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:48:01.755184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24d6eea-fe7d-4276-9652-55af9b278c14 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:01.768303 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24d6eea-fe7d-4276-9652-55af9b278c14 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:01.853847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca09959c-aa5f-493d-af84-7ae8cf7b6288 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:01.854526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca09959c-aa5f-493d-af84-7ae8cf7b6288 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:01.854841 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca09959c-aa5f-493d-af84-7ae8cf7b6288 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:01.855121 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca09959c-aa5f-493d-af84-7ae8cf7b6288 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:01.856128 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca09959c-aa5f-493d-af84-7ae8cf7b6288 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:01.858485 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca09959c-aa5f-493d-af84-7ae8cf7b6288 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:01.872455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1167/2327] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:01 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:48:01.877352 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04583e63-2ca1-4975-86bc-5d75bec5f857 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:01.919607 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04583e63-2ca1-4975-86bc-5d75bec5f857 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:01.936983 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ce5b69-163b-4981-b0f8-c2e165581926 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:01.937557 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ce5b69-163b-4981-b0f8-c2e165581926 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:01.938121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ce5b69-163b-4981-b0f8-c2e165581926 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:01.938730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ce5b69-163b-4981-b0f8-c2e165581926 admin admin] PATH_INFO: `/v3/projects/ccfb4ade5d154d08a2df76f8b19cb8bf/users/c61a1c1280b6421ea4ad99e38538b09b/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:01.943488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ce5b69-163b-4981-b0f8-c2e165581926 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ccfb4ade5d154d08a2df76f8b19cb8bf, user_id=c61a1c1280b6421ea4ad99e38538b09b, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:01.954161 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ce5b69-163b-4981-b0f8-c2e165581926 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:01.970197 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1161/2328] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:48:01 2022] PUT /identity/v3/projects/ccfb4ade5d154d08a2df76f8b19cb8bf/users/c61a1c1280b6421ea4ad99e38538b09b/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:01.980293 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f9fc3d-362b-4acd-93f9-786c62598cfc None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:01.980813 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f9fc3d-362b-4acd-93f9-786c62598cfc None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:01.984800 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f9fc3d-362b-4acd-93f9-786c62598cfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:02.019133 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-54f9fc3d-362b-4acd-93f9-786c62598cfc None None] MFA Rules not processed for user `c61a1c1280b6421ea4ad99e38538b09b`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:02.019700 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-54f9fc3d-362b-4acd-93f9-786c62598cfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:02.127275 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1168/2329] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:01 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:02.142440 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ce5b69-163b-4981-b0f8-c2e165581926 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:02.144848 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-78ce5b69-163b-4981-b0f8-c2e165581926 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:02.238010 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ce5b69-163b-4981-b0f8-c2e165581926 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:02.242418 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c63e9e-1201-4087-a94e-9cc8a55c80e1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:02.243080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c63e9e-1201-4087-a94e-9cc8a55c80e1 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:02.243878 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c63e9e-1201-4087-a94e-9cc8a55c80e1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:02.244156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c63e9e-1201-4087-a94e-9cc8a55c80e1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:02.255383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c63e9e-1201-4087-a94e-9cc8a55c80e1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:02.256622 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c63e9e-1201-4087-a94e-9cc8a55c80e1 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:02.268437 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1162/2330] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:48:02 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:48:02.912759 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:03.066095 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:03.285309 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba9aa38-fafc-4227-8852-8648d12e6c76 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:03.285598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba9aa38-fafc-4227-8852-8648d12e6c76 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:03.285873 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba9aa38-fafc-4227-8852-8648d12e6c76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:03.316234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-fba9aa38-fafc-4227-8852-8648d12e6c76 None None] MFA Rules not processed for user `4d46ec34caf74ea7b7c8f836c2f7ea6c`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:03.317587 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-fba9aa38-fafc-4227-8852-8648d12e6c76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:03.390795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1169/2331] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:03 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:03.394869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c63e9e-1201-4087-a94e-9cc8a55c80e1 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:03.396382 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-95c63e9e-1201-4087-a94e-9cc8a55c80e1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:03.474081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c63e9e-1201-4087-a94e-9cc8a55c80e1 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:03.479709 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29744f2-c999-4603-9026-a5660f9d27db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:03.480724 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29744f2-c999-4603-9026-a5660f9d27db admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:03.480996 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29744f2-c999-4603-9026-a5660f9d27db admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:03.481463 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29744f2-c999-4603-9026-a5660f9d27db admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:03.482099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29744f2-c999-4603-9026-a5660f9d27db admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:03.484638 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29744f2-c999-4603-9026-a5660f9d27db admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:03.593092 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1163/2332] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:48:03 2022] POST /identity/v3/projects => generated 383 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:48:03.597360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba9aa38-fafc-4227-8852-8648d12e6c76 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:03.611313 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba9aa38-fafc-4227-8852-8648d12e6c76 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:03.675903 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f19192-eeba-4c20-9e53-1c996a259423 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:03.676524 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f19192-eeba-4c20-9e53-1c996a259423 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:03.676977 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f19192-eeba-4c20-9e53-1c996a259423 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:03.677438 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f19192-eeba-4c20-9e53-1c996a259423 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:03.678086 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f19192-eeba-4c20-9e53-1c996a259423 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:03.680122 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f19192-eeba-4c20-9e53-1c996a259423 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:03.720698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29744f2-c999-4603-9026-a5660f9d27db admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:03.739213 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29744f2-c999-4603-9026-a5660f9d27db admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:03.745191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa1fdb3-7ec0-419c-9242-1e8e9854d296 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:03.748274 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa1fdb3-7ec0-419c-9242-1e8e9854d296 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:03.748274 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa1fdb3-7ec0-419c-9242-1e8e9854d296 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:03.748274 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa1fdb3-7ec0-419c-9242-1e8e9854d296 admin admin] PATH_INFO: `/v3/users/d4c198a8faba41c08ef15593a6f39642` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:03.748274 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa1fdb3-7ec0-419c-9242-1e8e9854d296 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d4c198a8faba41c08ef15593a6f39642)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:03.749532 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa1fdb3-7ec0-419c-9242-1e8e9854d296 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:03.773461 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1170/2333] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:48:03 2022] POST /identity/v3/users => generated 298 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:48:03.780247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f19192-eeba-4c20-9e53-1c996a259423 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:03.790441 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f19192-eeba-4c20-9e53-1c996a259423 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:03.795318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d96fc4-0869-4cb7-8a37-b8d48fbada03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:03.795958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d96fc4-0869-4cb7-8a37-b8d48fbada03 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:03.796245 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d96fc4-0869-4cb7-8a37-b8d48fbada03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:03.796523 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d96fc4-0869-4cb7-8a37-b8d48fbada03 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:03.797173 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d96fc4-0869-4cb7-8a37-b8d48fbada03 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:03.798902 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d96fc4-0869-4cb7-8a37-b8d48fbada03 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:03.809385 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1171/2334] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:48:03.813419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d96fc4-0869-4cb7-8a37-b8d48fbada03 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:03.823214 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d96fc4-0869-4cb7-8a37-b8d48fbada03 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:03.828671 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b38b131-a6b0-4e45-a1ce-3514a7bcabb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:03.829332 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b38b131-a6b0-4e45-a1ce-3514a7bcabb3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:03.829615 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b38b131-a6b0-4e45-a1ce-3514a7bcabb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:03.829890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b38b131-a6b0-4e45-a1ce-3514a7bcabb3 admin admin] PATH_INFO: `/v3/projects/ae05d51a1efc4c5398e9fa4ffeb7faa0/users/7166a00d45684b51aa90afa1b05ca7d8/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:03.847739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b38b131-a6b0-4e45-a1ce-3514a7bcabb3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae05d51a1efc4c5398e9fa4ffeb7faa0, user_id=7166a00d45684b51aa90afa1b05ca7d8, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:03.850630 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b38b131-a6b0-4e45-a1ce-3514a7bcabb3 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:03.942203 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1aa1fdb3-7ec0-419c-9242-1e8e9854d296 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd4c198a8faba41c08ef15593a6f39642'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:03.943167 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1aa1fdb3-7ec0-419c-9242-1e8e9854d296 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd4c198a8faba41c08ef15593a6f39642'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:03.943868 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1aa1fdb3-7ec0-419c-9242-1e8e9854d296 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd4c198a8faba41c08ef15593a6f39642'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:03.944923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1172/2335] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:48:03 2022] PUT /identity/v3/projects/ae05d51a1efc4c5398e9fa4ffeb7faa0/users/7166a00d45684b51aa90afa1b05ca7d8/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:48:03.948792 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b38b131-a6b0-4e45-a1ce-3514a7bcabb3 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:03.950641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7b38b131-a6b0-4e45-a1ce-3514a7bcabb3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:03.952774 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1aa1fdb3-7ec0-419c-9242-1e8e9854d296 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd4c198a8faba41c08ef15593a6f39642'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:04.020306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b38b131-a6b0-4e45-a1ce-3514a7bcabb3 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:04.025040 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c68807e-3945-4e11-a456-539dfef49939 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:04.025637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c68807e-3945-4e11-a456-539dfef49939 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:04.025917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c68807e-3945-4e11-a456-539dfef49939 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:04.026184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c68807e-3945-4e11-a456-539dfef49939 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:04.026856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c68807e-3945-4e11-a456-539dfef49939 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:04.028473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c68807e-3945-4e11-a456-539dfef49939 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:04.039256 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1173/2336] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:03 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:48:04.043244 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c68807e-3945-4e11-a456-539dfef49939 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:04.053162 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c68807e-3945-4e11-a456-539dfef49939 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:04.054063 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1aa1fdb3-7ec0-419c-9242-1e8e9854d296 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd4c198a8faba41c08ef15593a6f39642'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:04.058283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f785dbf0-30f2-4f62-9b3e-4150659255a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:04.059126 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f785dbf0-30f2-4f62-9b3e-4150659255a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:04.059311 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f785dbf0-30f2-4f62-9b3e-4150659255a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:04.059639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f785dbf0-30f2-4f62-9b3e-4150659255a9 admin admin] PATH_INFO: `/v3/projects/ae05d51a1efc4c5398e9fa4ffeb7faa0/users/7166a00d45684b51aa90afa1b05ca7d8/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:04.061648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f785dbf0-30f2-4f62-9b3e-4150659255a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae05d51a1efc4c5398e9fa4ffeb7faa0, user_id=7166a00d45684b51aa90afa1b05ca7d8, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:04.064650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f785dbf0-30f2-4f62-9b3e-4150659255a9 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:04.072648 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1164/2337] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:48:03 2022] DELETE /identity/v3/users/d4c198a8faba41c08ef15593a6f39642 => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:04.077751 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029d451e-235d-4c92-8009-f5fb0c529cdb None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:04.078033 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029d451e-235d-4c92-8009-f5fb0c529cdb None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:04.078303 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029d451e-235d-4c92-8009-f5fb0c529cdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:04.079699 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1174/2338] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:48:04 2022] PUT /identity/v3/projects/ae05d51a1efc4c5398e9fa4ffeb7faa0/users/7166a00d45684b51aa90afa1b05ca7d8/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:48:04.109150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-029d451e-235d-4c92-8009-f5fb0c529cdb None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:04.110575 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-029d451e-235d-4c92-8009-f5fb0c529cdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:04.178673 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1165/2339] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:48:04.186116 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f785dbf0-30f2-4f62-9b3e-4150659255a9 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:04.188286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-f785dbf0-30f2-4f62-9b3e-4150659255a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:04.267697 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f785dbf0-30f2-4f62-9b3e-4150659255a9 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:04.321298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8413d5-5e5e-4465-a9b9-3adba4efa91d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:04.321929 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8413d5-5e5e-4465-a9b9-3adba4efa91d service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:04.322243 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8413d5-5e5e-4465-a9b9-3adba4efa91d service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:04.322518 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8413d5-5e5e-4465-a9b9-3adba4efa91d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:04.333233 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8413d5-5e5e-4465-a9b9-3adba4efa91d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:04.334728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8413d5-5e5e-4465-a9b9-3adba4efa91d service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:04.345320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1175/2340] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:48:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:48:04.350584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:04.889599 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029d451e-235d-4c92-8009-f5fb0c529cdb None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:04.892545 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-029d451e-235d-4c92-8009-f5fb0c529cdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:04.917870 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029d451e-235d-4c92-8009-f5fb0c529cdb None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:04.929970 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3bf152-4102-4642-898f-975167753c94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:04.930945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3bf152-4102-4642-898f-975167753c94 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:04.932215 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3bf152-4102-4642-898f-975167753c94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:04.932600 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3bf152-4102-4642-898f-975167753c94 admin admin] PATH_INFO: `/v3/projects/b9df2c8224e94ba994c3971cf81dca4e` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:04.934002 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3bf152-4102-4642-898f-975167753c94 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b9df2c8224e94ba994c3971cf81dca4e)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:04.936477 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3bf152-4102-4642-898f-975167753c94 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:04.979002 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:05.086960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8a3bf152-4102-4642-898f-975167753c94 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b9df2c8224e94ba994c3971cf81dca4e', 'request_id': 'req-8a3bf152-4102-4642-898f-975167753c94'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:05.094917 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8a3bf152-4102-4642-898f-975167753c94 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b9df2c8224e94ba994c3971cf81dca4e', 'request_id': 'req-8a3bf152-4102-4642-898f-975167753c94'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:05.102335 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8a3bf152-4102-4642-898f-975167753c94 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b9df2c8224e94ba994c3971cf81dca4e', 'request_id': 'req-8a3bf152-4102-4642-898f-975167753c94'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:05.226829 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8a3bf152-4102-4642-898f-975167753c94 admin admin] The token cache is being invalidate because project b9df2c8224e94ba994c3971cf81dca4e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:48:05.227128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8a3bf152-4102-4642-898f-975167753c94 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:05.228239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8a3bf152-4102-4642-898f-975167753c94 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:05.230498 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1166/2341] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:48:04 2022] DELETE /identity/v3/projects/b9df2c8224e94ba994c3971cf81dca4e => generated 0 bytes in 341 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:05.235311 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8413d5-5e5e-4465-a9b9-3adba4efa91d service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:05.248072 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8413d5-5e5e-4465-a9b9-3adba4efa91d service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:05.330380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047ef81d-18ba-4db0-a4b5-d4cfea2b3f69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:05.331060 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047ef81d-18ba-4db0-a4b5-d4cfea2b3f69 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:05.331340 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047ef81d-18ba-4db0-a4b5-d4cfea2b3f69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:05.331619 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047ef81d-18ba-4db0-a4b5-d4cfea2b3f69 admin admin] PATH_INFO: `/v3/users/c61a1c1280b6421ea4ad99e38538b09b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:05.333061 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047ef81d-18ba-4db0-a4b5-d4cfea2b3f69 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c61a1c1280b6421ea4ad99e38538b09b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:05.334723 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047ef81d-18ba-4db0-a4b5-d4cfea2b3f69 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:05.386541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:05.444356 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:05.622578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-047ef81d-18ba-4db0-a4b5-d4cfea2b3f69 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c61a1c1280b6421ea4ad99e38538b09b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:05.637193 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-047ef81d-18ba-4db0-a4b5-d4cfea2b3f69 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c61a1c1280b6421ea4ad99e38538b09b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:05.638153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-047ef81d-18ba-4db0-a4b5-d4cfea2b3f69 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c61a1c1280b6421ea4ad99e38538b09b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:05.775859 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-047ef81d-18ba-4db0-a4b5-d4cfea2b3f69 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c61a1c1280b6421ea4ad99e38538b09b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:05.785670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-047ef81d-18ba-4db0-a4b5-d4cfea2b3f69 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c61a1c1280b6421ea4ad99e38538b09b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:05.827477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1176/2342] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:48:05 2022] DELETE /identity/v3/users/c61a1c1280b6421ea4ad99e38538b09b => generated 0 bytes in 564 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:06.923365 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3bf152-4102-4642-898f-975167753c94 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:06.925043 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8a3bf152-4102-4642-898f-975167753c94 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:06.956345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3bf152-4102-4642-898f-975167753c94 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:07.065610 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2294c2a-4617-4fe2-92bf-1e5e69d7452a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:07.066377 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2294c2a-4617-4fe2-92bf-1e5e69d7452a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:07.067631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2294c2a-4617-4fe2-92bf-1e5e69d7452a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:07.068011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2294c2a-4617-4fe2-92bf-1e5e69d7452a admin admin] PATH_INFO: `/v3/projects/ccfb4ade5d154d08a2df76f8b19cb8bf` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:07.072957 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2294c2a-4617-4fe2-92bf-1e5e69d7452a admin admin] RBAC: Authorizing `identity:delete_project(project_id=ccfb4ade5d154d08a2df76f8b19cb8bf)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:07.075079 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2294c2a-4617-4fe2-92bf-1e5e69d7452a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:07.236121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b2294c2a-4617-4fe2-92bf-1e5e69d7452a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ccfb4ade5d154d08a2df76f8b19cb8bf', 'request_id': 'req-b2294c2a-4617-4fe2-92bf-1e5e69d7452a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:07.259624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b2294c2a-4617-4fe2-92bf-1e5e69d7452a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ccfb4ade5d154d08a2df76f8b19cb8bf', 'request_id': 'req-b2294c2a-4617-4fe2-92bf-1e5e69d7452a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:07.277428 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b2294c2a-4617-4fe2-92bf-1e5e69d7452a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ccfb4ade5d154d08a2df76f8b19cb8bf', 'request_id': 'req-b2294c2a-4617-4fe2-92bf-1e5e69d7452a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:07.410309 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b2294c2a-4617-4fe2-92bf-1e5e69d7452a admin admin] The token cache is being invalidate because project ccfb4ade5d154d08a2df76f8b19cb8bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:48:07.410676 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b2294c2a-4617-4fe2-92bf-1e5e69d7452a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:07.411345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b2294c2a-4617-4fe2-92bf-1e5e69d7452a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:07.414632 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1167/2343] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:48:06 2022] DELETE /identity/v3/projects/ccfb4ade5d154d08a2df76f8b19cb8bf => generated 0 bytes in 491 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:07.418577 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee69459-5a72-4504-93fc-1aa8e232a040 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:07.419152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee69459-5a72-4504-93fc-1aa8e232a040 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:07.419260 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee69459-5a72-4504-93fc-1aa8e232a040 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:07.445137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-fee69459-5a72-4504-93fc-1aa8e232a040 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:07.446653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-fee69459-5a72-4504-93fc-1aa8e232a040 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:07.505274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:07.523465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1177/2344] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:07.535216 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38f8878-8974-42cf-ad60-cb85592d27dc None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:07.535216 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38f8878-8974-42cf-ad60-cb85592d27dc None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:07.535216 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38f8878-8974-42cf-ad60-cb85592d27dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:07.570816 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f38f8878-8974-42cf-ad60-cb85592d27dc None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:07.572173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f38f8878-8974-42cf-ad60-cb85592d27dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:07.580881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1168/2345] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:07.584754 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee69459-5a72-4504-93fc-1aa8e232a040 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:07.600191 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee69459-5a72-4504-93fc-1aa8e232a040 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:07.606638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd98964d-4dbf-42dd-878f-6cc51f0d27e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:07.607968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd98964d-4dbf-42dd-878f-6cc51f0d27e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:07.608355 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd98964d-4dbf-42dd-878f-6cc51f0d27e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:07.608728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd98964d-4dbf-42dd-878f-6cc51f0d27e2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:07.610299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd98964d-4dbf-42dd-878f-6cc51f0d27e2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:48:07.610742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd98964d-4dbf-42dd-878f-6cc51f0d27e2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:07.613390 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd98964d-4dbf-42dd-878f-6cc51f0d27e2 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:07.629017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1178/2346] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:48:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:48:07.633577 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38f8878-8974-42cf-ad60-cb85592d27dc None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:07.643533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38f8878-8974-42cf-ad60-cb85592d27dc None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:07.649780 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e480be90-b993-4bd1-809e-769782a61404 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:07.650512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e480be90-b993-4bd1-809e-769782a61404 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:07.651111 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e480be90-b993-4bd1-809e-769782a61404 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:07.651587 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e480be90-b993-4bd1-809e-769782a61404 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:07.652540 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e480be90-b993-4bd1-809e-769782a61404 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:07.654898 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e480be90-b993-4bd1-809e-769782a61404 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:07.782456 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1169/2347] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:48:07 2022] POST /identity/v3/projects => generated 387 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:48:07.786786 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd98964d-4dbf-42dd-878f-6cc51f0d27e2 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:07.798372 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd98964d-4dbf-42dd-878f-6cc51f0d27e2 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:07.863317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf9a4d6-101e-475c-9836-ae5e14264f8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:07.864053 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf9a4d6-101e-475c-9836-ae5e14264f8e admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:07.864441 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf9a4d6-101e-475c-9836-ae5e14264f8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:07.864840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf9a4d6-101e-475c-9836-ae5e14264f8e admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:07.865682 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf9a4d6-101e-475c-9836-ae5e14264f8e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:07.868846 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf9a4d6-101e-475c-9836-ae5e14264f8e admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:07.957907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1179/2348] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:48:07 2022] POST /identity/v3/users => generated 294 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:48:07.959202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e480be90-b993-4bd1-809e-769782a61404 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:07.965438 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:07.971698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e480be90-b993-4bd1-809e-769782a61404 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:07.977422 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb2c1fc-afb2-4f1d-97a6-2fe1c4af00dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:07.978068 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb2c1fc-afb2-4f1d-97a6-2fe1c4af00dd admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:07.978492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb2c1fc-afb2-4f1d-97a6-2fe1c4af00dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:07.978938 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb2c1fc-afb2-4f1d-97a6-2fe1c4af00dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:07.979833 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb2c1fc-afb2-4f1d-97a6-2fe1c4af00dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:07.981832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb2c1fc-afb2-4f1d-97a6-2fe1c4af00dd admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:07.995863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1170/2349] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:07 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:48:07.999993 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf9a4d6-101e-475c-9836-ae5e14264f8e admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:08.012480 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf9a4d6-101e-475c-9836-ae5e14264f8e admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:08.020345 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e89956e-9c94-4655-9023-b55aaac3a841 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:08.021624 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e89956e-9c94-4655-9023-b55aaac3a841 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:08.022124 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e89956e-9c94-4655-9023-b55aaac3a841 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:08.022719 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e89956e-9c94-4655-9023-b55aaac3a841 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:08.023663 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e89956e-9c94-4655-9023-b55aaac3a841 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:08.025581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e89956e-9c94-4655-9023-b55aaac3a841 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:08.040501 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1180/2350] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:07 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:48:08.044987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb2c1fc-afb2-4f1d-97a6-2fe1c4af00dd admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:08.056410 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb2c1fc-afb2-4f1d-97a6-2fe1c4af00dd admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:08.063113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f634a7d-bd8e-4434-ac7f-c3fbffc2a1fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:08.064009 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f634a7d-bd8e-4434-ac7f-c3fbffc2a1fe admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:08.064434 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f634a7d-bd8e-4434-ac7f-c3fbffc2a1fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:08.064838 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f634a7d-bd8e-4434-ac7f-c3fbffc2a1fe admin admin] PATH_INFO: `/v3/projects/26c6db332c174ae1baaba70e175deca6/users/a24bb98ca8e844f18a0c077c61b7f7d7/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:08.085763 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f634a7d-bd8e-4434-ac7f-c3fbffc2a1fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=26c6db332c174ae1baaba70e175deca6, user_id=a24bb98ca8e844f18a0c077c61b7f7d7, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:08.091120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f634a7d-bd8e-4434-ac7f-c3fbffc2a1fe admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:08.190823 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1171/2351] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:48:08 2022] PUT /identity/v3/projects/26c6db332c174ae1baaba70e175deca6/users/a24bb98ca8e844f18a0c077c61b7f7d7/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:08.192380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f113390-d507-41e7-9aaf-ffd0373226ea None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:08.192751 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f113390-d507-41e7-9aaf-ffd0373226ea None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:08.193105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f113390-d507-41e7-9aaf-ffd0373226ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:08.234822 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-1f113390-d507-41e7-9aaf-ffd0373226ea None None] MFA Rules not processed for user `a24bb98ca8e844f18a0c077c61b7f7d7`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:08.236631 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-1f113390-d507-41e7-9aaf-ffd0373226ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:08.350959 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f634a7d-bd8e-4434-ac7f-c3fbffc2a1fe admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:08.353874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1181/2352] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:08 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:08.451474 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f634a7d-bd8e-4434-ac7f-c3fbffc2a1fe admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:08.463999 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb6ea8e-a9ce-47af-acd8-1f83550fe83b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:08.466770 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb6ea8e-a9ce-47af-acd8-1f83550fe83b admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:08.467296 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb6ea8e-a9ce-47af-acd8-1f83550fe83b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:08.467781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb6ea8e-a9ce-47af-acd8-1f83550fe83b admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:08.469481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb6ea8e-a9ce-47af-acd8-1f83550fe83b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:08.474155 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb6ea8e-a9ce-47af-acd8-1f83550fe83b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:08.595840 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1172/2353] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:48:08 2022] POST /identity/v3/projects => generated 385 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:48:08.599824 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f113390-d507-41e7-9aaf-ffd0373226ea None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:08.611849 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f113390-d507-41e7-9aaf-ffd0373226ea None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:08.690269 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e909fb-3758-46ff-9b92-6e981266d318 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:08.691169 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e909fb-3758-46ff-9b92-6e981266d318 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:08.691631 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e909fb-3758-46ff-9b92-6e981266d318 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:08.692063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e909fb-3758-46ff-9b92-6e981266d318 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:08.697822 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e909fb-3758-46ff-9b92-6e981266d318 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:08.697822 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e909fb-3758-46ff-9b92-6e981266d318 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:08.776642 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1182/2354] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:48:08 2022] POST /identity/v3/users => generated 299 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:48:08.781058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb6ea8e-a9ce-47af-acd8-1f83550fe83b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:08.793735 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb6ea8e-a9ce-47af-acd8-1f83550fe83b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:08.809369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6616b056-1aac-4d2b-8479-a1b663674e25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:08.810964 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6616b056-1aac-4d2b-8479-a1b663674e25 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:08.812092 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6616b056-1aac-4d2b-8479-a1b663674e25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:08.812472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6616b056-1aac-4d2b-8479-a1b663674e25 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:08.813694 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6616b056-1aac-4d2b-8479-a1b663674e25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:08.818249 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6616b056-1aac-4d2b-8479-a1b663674e25 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:08.842538 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1173/2355] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:08 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:48:08.847178 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e909fb-3758-46ff-9b92-6e981266d318 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:08.860521 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e909fb-3758-46ff-9b92-6e981266d318 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:08.881194 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:08.882221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb3411e-fab2-4a71-bb1d-f199e56de072 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:08.883465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb3411e-fab2-4a71-bb1d-f199e56de072 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:08.883752 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb3411e-fab2-4a71-bb1d-f199e56de072 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:08.887238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb3411e-fab2-4a71-bb1d-f199e56de072 admin admin] PATH_INFO: `/v3/projects/17bfa06fa180441992ad20f3d523561f/users/96c52af2582a480e9d62bfddeb4c5ed8/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:08.905464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb3411e-fab2-4a71-bb1d-f199e56de072 admin admin] RBAC: Authorizing `identity:create_grant(project_id=17bfa06fa180441992ad20f3d523561f, user_id=96c52af2582a480e9d62bfddeb4c5ed8, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:08.908232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb3411e-fab2-4a71-bb1d-f199e56de072 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:09.031475 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1183/2356] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:48:08 2022] PUT /identity/v3/projects/17bfa06fa180441992ad20f3d523561f/users/96c52af2582a480e9d62bfddeb4c5ed8/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:09.035524 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6616b056-1aac-4d2b-8479-a1b663674e25 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:09.046340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6616b056-1aac-4d2b-8479-a1b663674e25 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:09.108972 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1130c0-0368-4947-82f6-9568db5b0681 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:09.109595 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1130c0-0368-4947-82f6-9568db5b0681 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:09.109873 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1130c0-0368-4947-82f6-9568db5b0681 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:09.110144 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1130c0-0368-4947-82f6-9568db5b0681 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:09.110865 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1130c0-0368-4947-82f6-9568db5b0681 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:09.112497 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1130c0-0368-4947-82f6-9568db5b0681 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:09.127605 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1174/2357] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:09 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:48:09.129423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb3411e-fab2-4a71-bb1d-f199e56de072 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:09.141011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb3411e-fab2-4a71-bb1d-f199e56de072 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:09.146126 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88a3427-3465-4c60-8ef9-811552bdee7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:09.151882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88a3427-3465-4c60-8ef9-811552bdee7b admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:09.151882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88a3427-3465-4c60-8ef9-811552bdee7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:09.151882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88a3427-3465-4c60-8ef9-811552bdee7b admin admin] PATH_INFO: `/v3/projects/17bfa06fa180441992ad20f3d523561f/users/96c52af2582a480e9d62bfddeb4c5ed8/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:09.151882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88a3427-3465-4c60-8ef9-811552bdee7b admin admin] RBAC: Authorizing `identity:create_grant(project_id=17bfa06fa180441992ad20f3d523561f, user_id=96c52af2582a480e9d62bfddeb4c5ed8, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:09.152203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88a3427-3465-4c60-8ef9-811552bdee7b admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:09.165125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1184/2358] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:48:09 2022] PUT /identity/v3/projects/17bfa06fa180441992ad20f3d523561f/users/96c52af2582a480e9d62bfddeb4c5ed8/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:48:09.175880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975703f7-90c7-48d6-94df-843a91c6ef6b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:09.176170 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975703f7-90c7-48d6-94df-843a91c6ef6b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:09.176446 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975703f7-90c7-48d6-94df-843a91c6ef6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:09.216170 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-975703f7-90c7-48d6-94df-843a91c6ef6b None None] MFA Rules not processed for user `96c52af2582a480e9d62bfddeb4c5ed8`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:09.217622 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-975703f7-90c7-48d6-94df-843a91c6ef6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:09.246661 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:09.297229 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1175/2359] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:09 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:09.305609 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88a3427-3465-4c60-8ef9-811552bdee7b admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:09.307906 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a88a3427-3465-4c60-8ef9-811552bdee7b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:09.384976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88a3427-3465-4c60-8ef9-811552bdee7b admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:09.388996 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d2d57a-746d-44d2-aecb-f6e68f389331 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:09.389780 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d2d57a-746d-44d2-aecb-f6e68f389331 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:09.390064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d2d57a-746d-44d2-aecb-f6e68f389331 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:09.390361 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d2d57a-746d-44d2-aecb-f6e68f389331 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:09.404716 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d2d57a-746d-44d2-aecb-f6e68f389331 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:09.408379 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d2d57a-746d-44d2-aecb-f6e68f389331 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:09.431378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1185/2360] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:48:09 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:48:09.509080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975703f7-90c7-48d6-94df-843a91c6ef6b None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:09.589081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975703f7-90c7-48d6-94df-843a91c6ef6b None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:09.594101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21699129-70f7-45ac-bac1-4d155561592b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:09.594863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21699129-70f7-45ac-bac1-4d155561592b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:09.595144 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21699129-70f7-45ac-bac1-4d155561592b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:09.595508 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21699129-70f7-45ac-bac1-4d155561592b admin admin] PATH_INFO: `/v3/users/a24bb98ca8e844f18a0c077c61b7f7d7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:09.596730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21699129-70f7-45ac-bac1-4d155561592b admin admin] RBAC: Authorizing `identity:delete_user(user_id=a24bb98ca8e844f18a0c077c61b7f7d7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:09.599262 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21699129-70f7-45ac-bac1-4d155561592b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:09.849458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-21699129-70f7-45ac-bac1-4d155561592b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a24bb98ca8e844f18a0c077c61b7f7d7'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:09.850821 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-21699129-70f7-45ac-bac1-4d155561592b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a24bb98ca8e844f18a0c077c61b7f7d7'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:09.851552 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-21699129-70f7-45ac-bac1-4d155561592b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a24bb98ca8e844f18a0c077c61b7f7d7'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:09.861194 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-21699129-70f7-45ac-bac1-4d155561592b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a24bb98ca8e844f18a0c077c61b7f7d7'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:09.927275 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-21699129-70f7-45ac-bac1-4d155561592b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a24bb98ca8e844f18a0c077c61b7f7d7'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:09.947959 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1176/2361] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:48:09 2022] DELETE /identity/v3/users/a24bb98ca8e844f18a0c077c61b7f7d7 => generated 0 bytes in 439 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:09.952750 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf42e7fd-00ea-48e2-9a44-132530891eaf None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:09.953339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf42e7fd-00ea-48e2-9a44-132530891eaf None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:09.953339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf42e7fd-00ea-48e2-9a44-132530891eaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:09.974466 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-cf42e7fd-00ea-48e2-9a44-132530891eaf None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:09.976074 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-cf42e7fd-00ea-48e2-9a44-132530891eaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:10.074793 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1186/2362] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:10.084425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21699129-70f7-45ac-bac1-4d155561592b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:10.086553 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-21699129-70f7-45ac-bac1-4d155561592b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:10.155490 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21699129-70f7-45ac-bac1-4d155561592b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:10.160910 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b0537f-37a7-4e4c-ac1f-23d78f24d28d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:10.161570 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b0537f-37a7-4e4c-ac1f-23d78f24d28d service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:10.162081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b0537f-37a7-4e4c-ac1f-23d78f24d28d service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:10.162611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b0537f-37a7-4e4c-ac1f-23d78f24d28d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:10.178922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b0537f-37a7-4e4c-ac1f-23d78f24d28d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:10.183412 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b0537f-37a7-4e4c-ac1f-23d78f24d28d service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:10.200505 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1177/2363] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:48:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:48:10.323926 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:10.947265 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f46b93-7db3-409c-842c-c20d935c49d5 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:10.947567 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f46b93-7db3-409c-842c-c20d935c49d5 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:10.947960 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f46b93-7db3-409c-842c-c20d935c49d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:10.993152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-91f46b93-7db3-409c-842c-c20d935c49d5 None None] MFA Rules not processed for user `7166a00d45684b51aa90afa1b05ca7d8`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:10.995176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-91f46b93-7db3-409c-842c-c20d935c49d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:11.038189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:11.084408 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1187/2364] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:10 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:11.098718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b0537f-37a7-4e4c-ac1f-23d78f24d28d service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:11.100374 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-40b0537f-37a7-4e4c-ac1f-23d78f24d28d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:11.169126 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b0537f-37a7-4e4c-ac1f-23d78f24d28d service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:11.173358 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa20346-6496-4e58-abad-8badfda6f830 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf907d4033064fb8a2707b1bbb0a180b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:11.174044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa20346-6496-4e58-abad-8badfda6f830 service glance] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:11.174396 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa20346-6496-4e58-abad-8badfda6f830 service glance] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:11.174708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa20346-6496-4e58-abad-8badfda6f830 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:11.180845 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4aa20346-6496-4e58-abad-8badfda6f830 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:11.253674 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa20346-6496-4e58-abad-8badfda6f830 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:11.255044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa20346-6496-4e58-abad-8badfda6f830 service glance] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:11.259483 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f46b93-7db3-409c-842c-c20d935c49d5 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:11.282133 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-91f46b93-7db3-409c-842c-c20d935c49d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:11.300512 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f46b93-7db3-409c-842c-c20d935c49d5 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:11.312998 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1178/2365] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:48:11 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:48:11.313490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb79843a-e8d0-4269-8201-f232ccf02ce5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:11.313490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb79843a-e8d0-4269-8201-f232ccf02ce5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:11.313490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb79843a-e8d0-4269-8201-f232ccf02ce5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:11.313490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb79843a-e8d0-4269-8201-f232ccf02ce5 admin admin] PATH_INFO: `/v3/projects/26c6db332c174ae1baaba70e175deca6` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:11.313777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb79843a-e8d0-4269-8201-f232ccf02ce5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=26c6db332c174ae1baaba70e175deca6)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:11.313777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb79843a-e8d0-4269-8201-f232ccf02ce5 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:11.491156 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-eb79843a-e8d0-4269-8201-f232ccf02ce5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '26c6db332c174ae1baaba70e175deca6', 'request_id': 'req-eb79843a-e8d0-4269-8201-f232ccf02ce5'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:11.494099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-eb79843a-e8d0-4269-8201-f232ccf02ce5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '26c6db332c174ae1baaba70e175deca6', 'request_id': 'req-eb79843a-e8d0-4269-8201-f232ccf02ce5'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:11.501108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-eb79843a-e8d0-4269-8201-f232ccf02ce5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '26c6db332c174ae1baaba70e175deca6', 'request_id': 'req-eb79843a-e8d0-4269-8201-f232ccf02ce5'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:11.597502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-eb79843a-e8d0-4269-8201-f232ccf02ce5 admin admin] The token cache is being invalidate because project 26c6db332c174ae1baaba70e175deca6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:48:11.603424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-eb79843a-e8d0-4269-8201-f232ccf02ce5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:11.603424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-eb79843a-e8d0-4269-8201-f232ccf02ce5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:11.603424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1188/2366] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:48:11 2022] DELETE /identity/v3/projects/26c6db332c174ae1baaba70e175deca6 => generated 0 bytes in 341 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:11.604555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa20346-6496-4e58-abad-8badfda6f830 service glance] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:11.615885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa20346-6496-4e58-abad-8badfda6f830 service glance] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:11.618779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ebf69c-cb51-4119-b61b-1e80f6b99864 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:11.619071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ebf69c-cb51-4119-b61b-1e80f6b99864 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:11.619344 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ebf69c-cb51-4119-b61b-1e80f6b99864 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:11.640555 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-c8ebf69c-cb51-4119-b61b-1e80f6b99864 None None] MFA Rules not processed for user `7d6e0545a40a436d8c63820b795551a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:11.641803 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c8ebf69c-cb51-4119-b61b-1e80f6b99864 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:11.677754 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e65459-5c59-498c-b343-b9ebb3300467 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:11.678370 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e65459-5c59-498c-b343-b9ebb3300467 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:11.678688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e65459-5c59-498c-b343-b9ebb3300467 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:11.678963 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e65459-5c59-498c-b343-b9ebb3300467 admin admin] PATH_INFO: `/v3/users/96c52af2582a480e9d62bfddeb4c5ed8` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:11.680080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e65459-5c59-498c-b343-b9ebb3300467 admin admin] RBAC: Authorizing `identity:delete_user(user_id=96c52af2582a480e9d62bfddeb4c5ed8)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:11.682041 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e65459-5c59-498c-b343-b9ebb3300467 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:11.698322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1189/2367] 10.222.0.39 () {58 vars in 1009 bytes} [Fri Sep 30 09:48:11 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:48:11.741289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ebf69c-cb51-4119-b61b-1e80f6b99864 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:11.742680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c8ebf69c-cb51-4119-b61b-1e80f6b99864 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:11.806625 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:11.812362 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ebf69c-cb51-4119-b61b-1e80f6b99864 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:11.817860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca52d7b2-de1d-4f25-8e85-27d86a27bbba service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e5ad921e6fa41c297696ec7279f0fc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:11.818471 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca52d7b2-de1d-4f25-8e85-27d86a27bbba service swift] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:11.818787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca52d7b2-de1d-4f25-8e85-27d86a27bbba service swift] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:11.819063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca52d7b2-de1d-4f25-8e85-27d86a27bbba service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:11.828850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca52d7b2-de1d-4f25-8e85-27d86a27bbba service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:11.830711 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca52d7b2-de1d-4f25-8e85-27d86a27bbba service swift] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:11.841956 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1190/2368] 10.222.0.39 () {60 vars in 1384 bytes} [Fri Sep 30 09:48:11 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 30 09:48:11.879788 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:11.885671 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c9e65459-5c59-498c-b343-b9ebb3300467 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '96c52af2582a480e9d62bfddeb4c5ed8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:11.888348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c9e65459-5c59-498c-b343-b9ebb3300467 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '96c52af2582a480e9d62bfddeb4c5ed8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:11.889106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c9e65459-5c59-498c-b343-b9ebb3300467 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '96c52af2582a480e9d62bfddeb4c5ed8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:11.897756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c9e65459-5c59-498c-b343-b9ebb3300467 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '96c52af2582a480e9d62bfddeb4c5ed8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:11.975048 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c9e65459-5c59-498c-b343-b9ebb3300467 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '96c52af2582a480e9d62bfddeb4c5ed8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:12.020527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1179/2369] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:48:11 2022] DELETE /identity/v3/users/96c52af2582a480e9d62bfddeb4c5ed8 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:12.586203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca52d7b2-de1d-4f25-8e85-27d86a27bbba service swift] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:12.587712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ca52d7b2-de1d-4f25-8e85-27d86a27bbba service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:12.634462 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca52d7b2-de1d-4f25-8e85-27d86a27bbba service swift] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:12.808980 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e2573d-1b1a-46ea-acc4-3c9f02d8941a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:12.810680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e2573d-1b1a-46ea-acc4-3c9f02d8941a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:12.810942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e2573d-1b1a-46ea-acc4-3c9f02d8941a admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:12.811221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e2573d-1b1a-46ea-acc4-3c9f02d8941a admin admin] PATH_INFO: `/v3/projects/17bfa06fa180441992ad20f3d523561f` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:12.815236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e2573d-1b1a-46ea-acc4-3c9f02d8941a admin admin] RBAC: Authorizing `identity:delete_project(project_id=17bfa06fa180441992ad20f3d523561f)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:12.820072 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e2573d-1b1a-46ea-acc4-3c9f02d8941a admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:13.360019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-65e2573d-1b1a-46ea-acc4-3c9f02d8941a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '17bfa06fa180441992ad20f3d523561f', 'request_id': 'req-65e2573d-1b1a-46ea-acc4-3c9f02d8941a'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:13.360019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-65e2573d-1b1a-46ea-acc4-3c9f02d8941a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '17bfa06fa180441992ad20f3d523561f', 'request_id': 'req-65e2573d-1b1a-46ea-acc4-3c9f02d8941a'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:13.360019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-65e2573d-1b1a-46ea-acc4-3c9f02d8941a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '17bfa06fa180441992ad20f3d523561f', 'request_id': 'req-65e2573d-1b1a-46ea-acc4-3c9f02d8941a'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:13.885250 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-65e2573d-1b1a-46ea-acc4-3c9f02d8941a admin admin] The token cache is being invalidate because project 17bfa06fa180441992ad20f3d523561f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:48:13.886245 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-65e2573d-1b1a-46ea-acc4-3c9f02d8941a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:13.887228 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-65e2573d-1b1a-46ea-acc4-3c9f02d8941a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:13.894996 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f9dc57-2f4d-4d3b-9973-eed8c05bab12 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:13.895950 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f9dc57-2f4d-4d3b-9973-eed8c05bab12 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:13.896479 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f9dc57-2f4d-4d3b-9973-eed8c05bab12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:13.899783 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1191/2370] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:48:12 2022] DELETE /identity/v3/projects/17bfa06fa180441992ad20f3d523561f => generated 0 bytes in 1304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:13.928972 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-61f9dc57-2f4d-4d3b-9973-eed8c05bab12 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:13.931324 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-61f9dc57-2f4d-4d3b-9973-eed8c05bab12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:14.056530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48c007b-7658-4274-bc4b-443f95e7be29 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:14.056937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48c007b-7658-4274-bc4b-443f95e7be29 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:14.057351 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48c007b-7658-4274-bc4b-443f95e7be29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:14.067973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1180/2371] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:14.068441 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f9dc57-2f4d-4d3b-9973-eed8c05bab12 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:14.089648 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-61f9dc57-2f4d-4d3b-9973-eed8c05bab12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:14.161951 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f9dc57-2f4d-4d3b-9973-eed8c05bab12 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:14.183495 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288e7881-377d-4be1-8619-74429b3bcd63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:14.185361 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288e7881-377d-4be1-8619-74429b3bcd63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:14.185876 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288e7881-377d-4be1-8619-74429b3bcd63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:14.186371 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288e7881-377d-4be1-8619-74429b3bcd63 admin admin] PATH_INFO: `/v3/users/cf23c11619e84dec8df2b424bd52ac07` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:14.188381 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288e7881-377d-4be1-8619-74429b3bcd63 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cf23c11619e84dec8df2b424bd52ac07)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:14.192989 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288e7881-377d-4be1-8619-74429b3bcd63 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:14.312843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-f48c007b-7658-4274-bc4b-443f95e7be29 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:14.313794 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-f48c007b-7658-4274-bc4b-443f95e7be29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:14.324428 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1192/2372] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 269 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:14.328054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48c007b-7658-4274-bc4b-443f95e7be29 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:14.375057 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48c007b-7658-4274-bc4b-443f95e7be29 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:14.402805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8ebd61-10f9-4c9c-8841-43fda82aa315 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:14.406292 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8ebd61-10f9-4c9c-8841-43fda82aa315 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:14.406574 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8ebd61-10f9-4c9c-8841-43fda82aa315 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:14.409449 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8ebd61-10f9-4c9c-8841-43fda82aa315 admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:14.409810 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8ebd61-10f9-4c9c-8841-43fda82aa315 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:48:14.410324 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8ebd61-10f9-4c9c-8841-43fda82aa315 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:14.435828 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8ebd61-10f9-4c9c-8841-43fda82aa315 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:14.477055 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-288e7881-377d-4be1-8619-74429b3bcd63 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cf23c11619e84dec8df2b424bd52ac07'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:14.479096 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-288e7881-377d-4be1-8619-74429b3bcd63 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cf23c11619e84dec8df2b424bd52ac07'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:14.480695 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-288e7881-377d-4be1-8619-74429b3bcd63 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cf23c11619e84dec8df2b424bd52ac07'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:14.513648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1193/2373] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:48:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:48:14.515941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8ebd61-10f9-4c9c-8841-43fda82aa315 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:14.518076 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ec8ebd61-10f9-4c9c-8841-43fda82aa315 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:14.527628 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-288e7881-377d-4be1-8619-74429b3bcd63 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cf23c11619e84dec8df2b424bd52ac07'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:14.557473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8ebd61-10f9-4c9c-8841-43fda82aa315 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:14.568236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3c8b11-5a73-4fe2-b5fa-8cb045d1b6ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:14.568856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3c8b11-5a73-4fe2-b5fa-8cb045d1b6ad admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:14.569251 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3c8b11-5a73-4fe2-b5fa-8cb045d1b6ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:14.569694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3c8b11-5a73-4fe2-b5fa-8cb045d1b6ad admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:14.570336 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3c8b11-5a73-4fe2-b5fa-8cb045d1b6ad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:14.573430 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3c8b11-5a73-4fe2-b5fa-8cb045d1b6ad admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:14.626043 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-288e7881-377d-4be1-8619-74429b3bcd63 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cf23c11619e84dec8df2b424bd52ac07'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:14.680511 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1181/2374] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:48:14 2022] DELETE /identity/v3/users/cf23c11619e84dec8df2b424bd52ac07 => generated 0 bytes in 613 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:14.687828 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8b8961-8716-488c-b9b2-4418d2939c75 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:14.688343 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8b8961-8716-488c-b9b2-4418d2939c75 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:14.688628 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8b8961-8716-488c-b9b2-4418d2939c75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:14.770376 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1194/2375] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:48:14 2022] POST /identity/v3/projects => generated 387 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:48:14.776435 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3c8b11-5a73-4fe2-b5fa-8cb045d1b6ad admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:14.792253 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7f8b8961-8716-488c-b9b2-4418d2939c75 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:14.793129 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7f8b8961-8716-488c-b9b2-4418d2939c75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:14.876109 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:14.985198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1182/2376] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 300 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:14.998156 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:14.998786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8b8961-8716-488c-b9b2-4418d2939c75 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:14.999454 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7f8b8961-8716-488c-b9b2-4418d2939c75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:15.071119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3c8b11-5a73-4fe2-b5fa-8cb045d1b6ad admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:15.109494 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd548598-b736-4a19-803d-e7771ce73fc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:15.112912 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd548598-b736-4a19-803d-e7771ce73fc0 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:15.114092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd548598-b736-4a19-803d-e7771ce73fc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:15.116071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd548598-b736-4a19-803d-e7771ce73fc0 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:15.118257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd548598-b736-4a19-803d-e7771ce73fc0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:15.127927 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd548598-b736-4a19-803d-e7771ce73fc0 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:15.134114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8b8961-8716-488c-b9b2-4418d2939c75 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:15.157339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d951b072-10d4-43e8-96da-584588d36f4e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:15.158312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d951b072-10d4-43e8-96da-584588d36f4e service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:15.159059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d951b072-10d4-43e8-96da-584588d36f4e service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:15.159529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d951b072-10d4-43e8-96da-584588d36f4e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:15.177572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d951b072-10d4-43e8-96da-584588d36f4e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:15.182641 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d951b072-10d4-43e8-96da-584588d36f4e service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:15.222837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1183/2377] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:48:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:48:15.279946 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1195/2378] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:48:14 2022] POST /identity/v3/users => generated 294 bytes in 505 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:48:15.283428 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d951b072-10d4-43e8-96da-584588d36f4e service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:15.309271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d951b072-10d4-43e8-96da-584588d36f4e service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:15.329067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf813ff-c05a-4a59-9464-c1c14ea120f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:15.333983 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf813ff-c05a-4a59-9464-c1c14ea120f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:15.337754 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf813ff-c05a-4a59-9464-c1c14ea120f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:15.338286 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf813ff-c05a-4a59-9464-c1c14ea120f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:15.339523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf813ff-c05a-4a59-9464-c1c14ea120f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:15.360575 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf813ff-c05a-4a59-9464-c1c14ea120f6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:15.429427 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1184/2379] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:15 2022] GET /identity/v3/roles => generated 1420 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:48:15.433969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd548598-b736-4a19-803d-e7771ce73fc0 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:15.448439 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd548598-b736-4a19-803d-e7771ce73fc0 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:15.462446 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3736dce1-db0d-43b0-92c3-61e47f59d72e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:15.466008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3736dce1-db0d-43b0-92c3-61e47f59d72e admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:15.466311 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3736dce1-db0d-43b0-92c3-61e47f59d72e admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:15.467126 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3736dce1-db0d-43b0-92c3-61e47f59d72e admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:15.467811 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3736dce1-db0d-43b0-92c3-61e47f59d72e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:15.479679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3736dce1-db0d-43b0-92c3-61e47f59d72e admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:15.500269 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1196/2380] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:15 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:48:15.504992 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf813ff-c05a-4a59-9464-c1c14ea120f6 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:15.522243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf813ff-c05a-4a59-9464-c1c14ea120f6 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:15.538959 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0eb7761-0693-471e-9c16-6e3eb448947d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:15.541649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eb7761-0693-471e-9c16-6e3eb448947d admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:15.542015 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eb7761-0693-471e-9c16-6e3eb448947d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:15.542298 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eb7761-0693-471e-9c16-6e3eb448947d admin admin] PATH_INFO: `/v3/projects/2ff64158d3244f299d9a50abd9af4be4/users/4a6e55f8848d426f982a79b93900222e/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:15.577466 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0eb7761-0693-471e-9c16-6e3eb448947d admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ff64158d3244f299d9a50abd9af4be4, user_id=4a6e55f8848d426f982a79b93900222e, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:15.588782 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0eb7761-0693-471e-9c16-6e3eb448947d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:15.698828 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1185/2381] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:48:15 2022] PUT /identity/v3/projects/2ff64158d3244f299d9a50abd9af4be4/users/4a6e55f8848d426f982a79b93900222e/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:15.710269 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950d605c-4aa0-46d7-a81c-0723701fd49e None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:15.710919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950d605c-4aa0-46d7-a81c-0723701fd49e None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:15.711352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950d605c-4aa0-46d7-a81c-0723701fd49e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:15.760769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-950d605c-4aa0-46d7-a81c-0723701fd49e None None] MFA Rules not processed for user `4a6e55f8848d426f982a79b93900222e`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:15.762461 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-950d605c-4aa0-46d7-a81c-0723701fd49e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:15.918313 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1197/2382] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:15 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:15.922730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0eb7761-0693-471e-9c16-6e3eb448947d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:16.046073 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0eb7761-0693-471e-9c16-6e3eb448947d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:16.053299 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398ffcb5-f5d9-4903-bae0-906317fe0c9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:16.054160 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398ffcb5-f5d9-4903-bae0-906317fe0c9b admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:16.054685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398ffcb5-f5d9-4903-bae0-906317fe0c9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:16.055178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398ffcb5-f5d9-4903-bae0-906317fe0c9b admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:16.056236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398ffcb5-f5d9-4903-bae0-906317fe0c9b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:16.058467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398ffcb5-f5d9-4903-bae0-906317fe0c9b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:16.186146 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950d605c-4aa0-46d7-a81c-0723701fd49e None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:16.188750 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-950d605c-4aa0-46d7-a81c-0723701fd49e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:16.203546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1186/2383] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:48:15 2022] POST /identity/v3/projects => generated 389 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:48:16.213313 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398ffcb5-f5d9-4903-bae0-906317fe0c9b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:16.219761 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950d605c-4aa0-46d7-a81c-0723701fd49e None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:16.301447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:16.395527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf434778-f142-4df4-a75e-eed1f5300616 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:16.396446 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf434778-f142-4df4-a75e-eed1f5300616 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:16.396918 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf434778-f142-4df4-a75e-eed1f5300616 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:16.397456 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf434778-f142-4df4-a75e-eed1f5300616 admin admin] PATH_INFO: `/v3/projects/d6413896cf7941dc9f287e9e9ef42498` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:16.401509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf434778-f142-4df4-a75e-eed1f5300616 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d6413896cf7941dc9f287e9e9ef42498)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:16.404138 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf434778-f142-4df4-a75e-eed1f5300616 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:16.472102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398ffcb5-f5d9-4903-bae0-906317fe0c9b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:16.499217 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980d49e0-dd86-4cf2-a49a-e7324430d004 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:16.502149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980d49e0-dd86-4cf2-a49a-e7324430d004 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:16.502149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980d49e0-dd86-4cf2-a49a-e7324430d004 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:16.502149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980d49e0-dd86-4cf2-a49a-e7324430d004 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:16.506232 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980d49e0-dd86-4cf2-a49a-e7324430d004 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:16.511211 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980d49e0-dd86-4cf2-a49a-e7324430d004 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:16.710464 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1187/2384] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:48:16 2022] POST /identity/v3/users => generated 301 bytes in 498 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:48:16.711028 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:16.713026 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-bf434778-f142-4df4-a75e-eed1f5300616 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd6413896cf7941dc9f287e9e9ef42498', 'request_id': 'req-bf434778-f142-4df4-a75e-eed1f5300616'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:16.723296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-bf434778-f142-4df4-a75e-eed1f5300616 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd6413896cf7941dc9f287e9e9ef42498', 'request_id': 'req-bf434778-f142-4df4-a75e-eed1f5300616'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:16.735280 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980d49e0-dd86-4cf2-a49a-e7324430d004 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:16.750369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-bf434778-f142-4df4-a75e-eed1f5300616 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd6413896cf7941dc9f287e9e9ef42498', 'request_id': 'req-bf434778-f142-4df4-a75e-eed1f5300616'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:16.751026 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980d49e0-dd86-4cf2-a49a-e7324430d004 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:16.838324 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-bf434778-f142-4df4-a75e-eed1f5300616 admin admin] The token cache is being invalidate because project d6413896cf7941dc9f287e9e9ef42498 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:48:16.838649 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-bf434778-f142-4df4-a75e-eed1f5300616 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:16.839330 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-bf434778-f142-4df4-a75e-eed1f5300616 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:16.852376 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1198/2385] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:48:16 2022] DELETE /identity/v3/projects/d6413896cf7941dc9f287e9e9ef42498 => generated 0 bytes in 656 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:16.921472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4b4340-1b2a-4e41-aee9-441e93f86bd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:16.921472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4b4340-1b2a-4e41-aee9-441e93f86bd0 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:16.921472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4b4340-1b2a-4e41-aee9-441e93f86bd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:16.921472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4b4340-1b2a-4e41-aee9-441e93f86bd0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:16.926517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4b4340-1b2a-4e41-aee9-441e93f86bd0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:16.926517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4b4340-1b2a-4e41-aee9-441e93f86bd0 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:16.955916 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1188/2386] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:16 2022] GET /identity/v3/roles => generated 1420 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:48:16.960329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf434778-f142-4df4-a75e-eed1f5300616 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:16.960468 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-bf434778-f142-4df4-a75e-eed1f5300616 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:16.993841 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf434778-f142-4df4-a75e-eed1f5300616 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:17.029019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80089d7-acde-4ab1-a9a3-82ac5a4b2427 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:17.034126 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80089d7-acde-4ab1-a9a3-82ac5a4b2427 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:17.039899 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80089d7-acde-4ab1-a9a3-82ac5a4b2427 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:17.041106 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80089d7-acde-4ab1-a9a3-82ac5a4b2427 admin admin] PATH_INFO: `/v3/projects/587193b710ea403bade7288be6ec5f84/users/4c1db8f009934e0890efa85d2ff97463/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:17.107617 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80089d7-acde-4ab1-a9a3-82ac5a4b2427 admin admin] RBAC: Authorizing `identity:create_grant(project_id=587193b710ea403bade7288be6ec5f84, user_id=4c1db8f009934e0890efa85d2ff97463, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:17.110585 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80089d7-acde-4ab1-a9a3-82ac5a4b2427 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:17.213070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1199/2387] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:48:16 2022] PUT /identity/v3/projects/587193b710ea403bade7288be6ec5f84/users/4c1db8f009934e0890efa85d2ff97463/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:17.215938 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4b4340-1b2a-4e41-aee9-441e93f86bd0 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:17.229573 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4b4340-1b2a-4e41-aee9-441e93f86bd0 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:17.360524 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac25754-16ba-432b-beb0-2543d1d4908a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:17.361276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac25754-16ba-432b-beb0-2543d1d4908a admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:17.361617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac25754-16ba-432b-beb0-2543d1d4908a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:17.362116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac25754-16ba-432b-beb0-2543d1d4908a admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:17.362811 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac25754-16ba-432b-beb0-2543d1d4908a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:17.364459 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac25754-16ba-432b-beb0-2543d1d4908a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:17.392060 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80089d7-acde-4ab1-a9a3-82ac5a4b2427 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:17.394144 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1189/2388] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:17 2022] GET /identity/v3/roles => generated 1420 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:48:17.413622 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80089d7-acde-4ab1-a9a3-82ac5a4b2427 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:17.458896 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5997452a-24f9-4cf7-93dc-e9d1e07a4540 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:17.458896 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5997452a-24f9-4cf7-93dc-e9d1e07a4540 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:17.458896 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5997452a-24f9-4cf7-93dc-e9d1e07a4540 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:17.458896 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5997452a-24f9-4cf7-93dc-e9d1e07a4540 admin admin] PATH_INFO: `/v3/projects/587193b710ea403bade7288be6ec5f84/users/4c1db8f009934e0890efa85d2ff97463/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:17.460532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5997452a-24f9-4cf7-93dc-e9d1e07a4540 admin admin] RBAC: Authorizing `identity:create_grant(project_id=587193b710ea403bade7288be6ec5f84, user_id=4c1db8f009934e0890efa85d2ff97463, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:17.460532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5997452a-24f9-4cf7-93dc-e9d1e07a4540 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:17.488576 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1200/2389] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:48:17 2022] PUT /identity/v3/projects/587193b710ea403bade7288be6ec5f84/users/4c1db8f009934e0890efa85d2ff97463/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:17.506663 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630eceba-0a82-4f19-862a-ad14097efa8a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:17.507018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630eceba-0a82-4f19-862a-ad14097efa8a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:17.507386 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630eceba-0a82-4f19-862a-ad14097efa8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:17.560302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-630eceba-0a82-4f19-862a-ad14097efa8a None None] MFA Rules not processed for user `4c1db8f009934e0890efa85d2ff97463`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:17.561536 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-630eceba-0a82-4f19-862a-ad14097efa8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:17.630065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:17.673727 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1190/2390] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:17 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:17.690964 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5997452a-24f9-4cf7-93dc-e9d1e07a4540 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:17.690964 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-5997452a-24f9-4cf7-93dc-e9d1e07a4540 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:17.752298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5997452a-24f9-4cf7-93dc-e9d1e07a4540 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:17.756043 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d181f0a6-9daf-4209-9e32-b1057a33c65d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:17.756656 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d181f0a6-9daf-4209-9e32-b1057a33c65d service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:17.756938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d181f0a6-9daf-4209-9e32-b1057a33c65d service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:17.757220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d181f0a6-9daf-4209-9e32-b1057a33c65d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:17.767113 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d181f0a6-9daf-4209-9e32-b1057a33c65d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:17.768581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d181f0a6-9daf-4209-9e32-b1057a33c65d service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:17.781681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1201/2391] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:48:17 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:48:18.148184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630eceba-0a82-4f19-862a-ad14097efa8a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:18.290488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630eceba-0a82-4f19-862a-ad14097efa8a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:18.302260 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2d5add-d286-4c38-8c31-17a41c93fddc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:18.303029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2d5add-d286-4c38-8c31-17a41c93fddc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:18.303316 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2d5add-d286-4c38-8c31-17a41c93fddc admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:18.303596 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2d5add-d286-4c38-8c31-17a41c93fddc admin admin] PATH_INFO: `/v3/users/4a6e55f8848d426f982a79b93900222e` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:18.305057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2d5add-d286-4c38-8c31-17a41c93fddc admin admin] RBAC: Authorizing `identity:delete_user(user_id=4a6e55f8848d426f982a79b93900222e)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:18.319345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2d5add-d286-4c38-8c31-17a41c93fddc admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:18.610756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8a2d5add-d286-4c38-8c31-17a41c93fddc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4a6e55f8848d426f982a79b93900222e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:18.612750 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8a2d5add-d286-4c38-8c31-17a41c93fddc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4a6e55f8848d426f982a79b93900222e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:18.616959 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8a2d5add-d286-4c38-8c31-17a41c93fddc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4a6e55f8848d426f982a79b93900222e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:18.629308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8a2d5add-d286-4c38-8c31-17a41c93fddc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4a6e55f8848d426f982a79b93900222e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:18.756426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8a2d5add-d286-4c38-8c31-17a41c93fddc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4a6e55f8848d426f982a79b93900222e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:18.831758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1191/2392] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:48:18 2022] DELETE /identity/v3/users/4a6e55f8848d426f982a79b93900222e => generated 0 bytes in 684 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:18.835960 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edbd29c-795e-43f5-b4c4-bc4365b2e6d9 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:18.836499 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edbd29c-795e-43f5-b4c4-bc4365b2e6d9 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:18.843575 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edbd29c-795e-43f5-b4c4-bc4365b2e6d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:18.900587 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-3edbd29c-795e-43f5-b4c4-bc4365b2e6d9 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:18.902847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-3edbd29c-795e-43f5-b4c4-bc4365b2e6d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:19.009418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1202/2393] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:19.019291 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2d5add-d286-4c38-8c31-17a41c93fddc admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:19.021217 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8a2d5add-d286-4c38-8c31-17a41c93fddc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:19.135196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2d5add-d286-4c38-8c31-17a41c93fddc admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:19.139830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579c6637-5594-4242-9307-689ca05e333b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:19.140458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579c6637-5594-4242-9307-689ca05e333b service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:19.140735 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579c6637-5594-4242-9307-689ca05e333b service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:19.141007 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579c6637-5594-4242-9307-689ca05e333b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:19.157186 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:19.171116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-579c6637-5594-4242-9307-689ca05e333b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:19.175387 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-579c6637-5594-4242-9307-689ca05e333b service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:19.195762 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1192/2394] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:48:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:48:19.781926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edbd29c-795e-43f5-b4c4-bc4365b2e6d9 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:19.783517 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-3edbd29c-795e-43f5-b4c4-bc4365b2e6d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:19.822833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edbd29c-795e-43f5-b4c4-bc4365b2e6d9 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:19.827809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62c233c-c1f4-4cfc-9e13-426344c0653c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:19.828556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62c233c-c1f4-4cfc-9e13-426344c0653c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:19.828931 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62c233c-c1f4-4cfc-9e13-426344c0653c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:19.829399 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62c233c-c1f4-4cfc-9e13-426344c0653c admin admin] PATH_INFO: `/v3/projects/2ff64158d3244f299d9a50abd9af4be4` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:19.830830 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62c233c-c1f4-4cfc-9e13-426344c0653c admin admin] RBAC: Authorizing `identity:delete_project(project_id=2ff64158d3244f299d9a50abd9af4be4)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:19.833815 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62c233c-c1f4-4cfc-9e13-426344c0653c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:19.928163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=91,keystone:DELETE=6,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:19.939031 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e62c233c-c1f4-4cfc-9e13-426344c0653c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2ff64158d3244f299d9a50abd9af4be4', 'request_id': 'req-e62c233c-c1f4-4cfc-9e13-426344c0653c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:19.950155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e62c233c-c1f4-4cfc-9e13-426344c0653c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2ff64158d3244f299d9a50abd9af4be4', 'request_id': 'req-e62c233c-c1f4-4cfc-9e13-426344c0653c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:19.958416 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e62c233c-c1f4-4cfc-9e13-426344c0653c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2ff64158d3244f299d9a50abd9af4be4', 'request_id': 'req-e62c233c-c1f4-4cfc-9e13-426344c0653c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:20.055358 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e62c233c-c1f4-4cfc-9e13-426344c0653c admin admin] The token cache is being invalidate because project 2ff64158d3244f299d9a50abd9af4be4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:48:20.055907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e62c233c-c1f4-4cfc-9e13-426344c0653c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:20.056725 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e62c233c-c1f4-4cfc-9e13-426344c0653c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:20.061446 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1203/2395] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:48:19 2022] DELETE /identity/v3/projects/2ff64158d3244f299d9a50abd9af4be4 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:20.065731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579c6637-5594-4242-9307-689ca05e333b service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:20.079350 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579c6637-5594-4242-9307-689ca05e333b service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:20.154764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cfae43-8f44-48d0-8fe5-d898f7d10a3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:20.155403 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cfae43-8f44-48d0-8fe5-d898f7d10a3f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:20.155685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cfae43-8f44-48d0-8fe5-d898f7d10a3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:20.155977 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cfae43-8f44-48d0-8fe5-d898f7d10a3f admin admin] PATH_INFO: `/v3/users/4c1db8f009934e0890efa85d2ff97463` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:20.157138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cfae43-8f44-48d0-8fe5-d898f7d10a3f admin admin] RBAC: Authorizing `identity:delete_user(user_id=4c1db8f009934e0890efa85d2ff97463)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:20.162699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cfae43-8f44-48d0-8fe5-d898f7d10a3f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:20.335432 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d1cfae43-8f44-48d0-8fe5-d898f7d10a3f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4c1db8f009934e0890efa85d2ff97463'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:20.336432 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d1cfae43-8f44-48d0-8fe5-d898f7d10a3f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4c1db8f009934e0890efa85d2ff97463'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:20.337297 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d1cfae43-8f44-48d0-8fe5-d898f7d10a3f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4c1db8f009934e0890efa85d2ff97463'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:20.349048 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d1cfae43-8f44-48d0-8fe5-d898f7d10a3f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4c1db8f009934e0890efa85d2ff97463'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:20.441316 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d1cfae43-8f44-48d0-8fe5-d898f7d10a3f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4c1db8f009934e0890efa85d2ff97463'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:20.468091 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1193/2396] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:48:20 2022] DELETE /identity/v3/users/4c1db8f009934e0890efa85d2ff97463 => generated 0 bytes in 399 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:20.940113 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62c233c-c1f4-4cfc-9e13-426344c0653c admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:20.945997 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-e62c233c-c1f4-4cfc-9e13-426344c0653c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:20.987984 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62c233c-c1f4-4cfc-9e13-426344c0653c admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:21.071513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5eb645-0577-47fc-9c38-580fcf9cc0f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:21.072282 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5eb645-0577-47fc-9c38-580fcf9cc0f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:21.072797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5eb645-0577-47fc-9c38-580fcf9cc0f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:21.073291 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5eb645-0577-47fc-9c38-580fcf9cc0f6 admin admin] PATH_INFO: `/v3/projects/587193b710ea403bade7288be6ec5f84` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:21.074642 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5eb645-0577-47fc-9c38-580fcf9cc0f6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=587193b710ea403bade7288be6ec5f84)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:21.077259 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5eb645-0577-47fc-9c38-580fcf9cc0f6 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:21.233174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-4e5eb645-0577-47fc-9c38-580fcf9cc0f6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '587193b710ea403bade7288be6ec5f84', 'request_id': 'req-4e5eb645-0577-47fc-9c38-580fcf9cc0f6'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:21.240978 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-4e5eb645-0577-47fc-9c38-580fcf9cc0f6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '587193b710ea403bade7288be6ec5f84', 'request_id': 'req-4e5eb645-0577-47fc-9c38-580fcf9cc0f6'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:21.248981 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-4e5eb645-0577-47fc-9c38-580fcf9cc0f6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '587193b710ea403bade7288be6ec5f84', 'request_id': 'req-4e5eb645-0577-47fc-9c38-580fcf9cc0f6'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:21.323071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-4e5eb645-0577-47fc-9c38-580fcf9cc0f6 admin admin] The token cache is being invalidate because project 587193b710ea403bade7288be6ec5f84 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:48:21.323404 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-4e5eb645-0577-47fc-9c38-580fcf9cc0f6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:21.324258 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-4e5eb645-0577-47fc-9c38-580fcf9cc0f6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:21.327285 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1204/2397] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:48:20 2022] DELETE /identity/v3/projects/587193b710ea403bade7288be6ec5f84 => generated 0 bytes in 387 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:21.334306 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c1a8f4-a1cc-41a2-aac8-555d0f73509a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:21.335252 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c1a8f4-a1cc-41a2-aac8-555d0f73509a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:21.336356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c1a8f4-a1cc-41a2-aac8-555d0f73509a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:21.357755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d4c1a8f4-a1cc-41a2-aac8-555d0f73509a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:21.359081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d4c1a8f4-a1cc-41a2-aac8-555d0f73509a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:21.421431 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1194/2398] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:21.429278 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dcbad6-6e68-4b91-b939-dd31575c69c0 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:21.430318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dcbad6-6e68-4b91-b939-dd31575c69c0 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:21.430318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dcbad6-6e68-4b91-b939-dd31575c69c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:21.453174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-80dcbad6-6e68-4b91-b939-dd31575c69c0 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:21.454467 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-80dcbad6-6e68-4b91-b939-dd31575c69c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:21.462452 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1205/2399] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:21.466741 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c1a8f4-a1cc-41a2-aac8-555d0f73509a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:21.476277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c1a8f4-a1cc-41a2-aac8-555d0f73509a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:21.480807 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0437f9be-2088-4989-9b1a-019a5aa851ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:21.481438 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0437f9be-2088-4989-9b1a-019a5aa851ed admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:21.481718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0437f9be-2088-4989-9b1a-019a5aa851ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:21.481992 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0437f9be-2088-4989-9b1a-019a5aa851ed admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:21.482664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0437f9be-2088-4989-9b1a-019a5aa851ed admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:48:21.482971 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0437f9be-2088-4989-9b1a-019a5aa851ed admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:21.484564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0437f9be-2088-4989-9b1a-019a5aa851ed admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:21.493364 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:21.497084 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1195/2400] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:48:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:48:21.501216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80dcbad6-6e68-4b91-b939-dd31575c69c0 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:21.510253 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80dcbad6-6e68-4b91-b939-dd31575c69c0 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:21.515488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2ba7e7-acd0-473f-a838-7ea1428e9078 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:21.521099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2ba7e7-acd0-473f-a838-7ea1428e9078 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:21.521099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2ba7e7-acd0-473f-a838-7ea1428e9078 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:21.521099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2ba7e7-acd0-473f-a838-7ea1428e9078 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:21.521099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2ba7e7-acd0-473f-a838-7ea1428e9078 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:21.521099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2ba7e7-acd0-473f-a838-7ea1428e9078 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:21.872746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1206/2401] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:48:21 2022] POST /identity/v3/projects => generated 383 bytes in 372 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:48:21.876669 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0437f9be-2088-4989-9b1a-019a5aa851ed admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:21.887093 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0437f9be-2088-4989-9b1a-019a5aa851ed admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:21.947428 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a30f095-70bc-48f3-8797-9511658e8da6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:21.948076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a30f095-70bc-48f3-8797-9511658e8da6 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:21.948320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a30f095-70bc-48f3-8797-9511658e8da6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:21.948594 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a30f095-70bc-48f3-8797-9511658e8da6 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:21.949226 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a30f095-70bc-48f3-8797-9511658e8da6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:21.951407 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a30f095-70bc-48f3-8797-9511658e8da6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:22.047992 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1196/2402] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:48:21 2022] POST /identity/v3/users => generated 292 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:48:22.051720 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2ba7e7-acd0-473f-a838-7ea1428e9078 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:22.063809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2ba7e7-acd0-473f-a838-7ea1428e9078 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:22.068992 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0cb499-b3f0-4f3a-8836-f680b8a7716d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:22.069821 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0cb499-b3f0-4f3a-8836-f680b8a7716d admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:22.074830 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0cb499-b3f0-4f3a-8836-f680b8a7716d admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:22.075159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0cb499-b3f0-4f3a-8836-f680b8a7716d admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:22.075856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0cb499-b3f0-4f3a-8836-f680b8a7716d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:22.079316 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0cb499-b3f0-4f3a-8836-f680b8a7716d admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:22.090401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1207/2403] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:22 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:48:22.094214 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a30f095-70bc-48f3-8797-9511658e8da6 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:22.105231 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a30f095-70bc-48f3-8797-9511658e8da6 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:22.111315 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef7f08d-edf2-4006-83d5-d7b918bb4350 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:22.112685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef7f08d-edf2-4006-83d5-d7b918bb4350 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:22.112965 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef7f08d-edf2-4006-83d5-d7b918bb4350 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:22.113235 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef7f08d-edf2-4006-83d5-d7b918bb4350 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:22.114057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef7f08d-edf2-4006-83d5-d7b918bb4350 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:22.115962 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef7f08d-edf2-4006-83d5-d7b918bb4350 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:22.137720 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1197/2404] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:22 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:48:22.138699 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0cb499-b3f0-4f3a-8836-f680b8a7716d admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:22.150581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0cb499-b3f0-4f3a-8836-f680b8a7716d admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:22.159162 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bab97e5-a466-4d30-bfa1-436fc23a8bc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:22.160095 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bab97e5-a466-4d30-bfa1-436fc23a8bc9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:22.160586 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bab97e5-a466-4d30-bfa1-436fc23a8bc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:22.160866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bab97e5-a466-4d30-bfa1-436fc23a8bc9 admin admin] PATH_INFO: `/v3/projects/7e897f8836fe4715ab1b8ec5abf574d7/users/44043fefdb034ee7adef616458c4ca36/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:22.174422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:22.182779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bab97e5-a466-4d30-bfa1-436fc23a8bc9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e897f8836fe4715ab1b8ec5abf574d7, user_id=44043fefdb034ee7adef616458c4ca36, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:22.185685 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bab97e5-a466-4d30-bfa1-436fc23a8bc9 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:22.267035 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c564cd1-0e52-456b-b293-122f6f41d315 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:22.267411 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c564cd1-0e52-456b-b293-122f6f41d315 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:22.267759 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c564cd1-0e52-456b-b293-122f6f41d315 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:22.280329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1208/2405] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:48:22 2022] PUT /identity/v3/projects/7e897f8836fe4715ab1b8ec5abf574d7/users/44043fefdb034ee7adef616458c4ca36/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:48:22.327824 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9c564cd1-0e52-456b-b293-122f6f41d315 None None] MFA Rules not processed for user `44043fefdb034ee7adef616458c4ca36`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:22.329645 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9c564cd1-0e52-456b-b293-122f6f41d315 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:22.437268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1198/2406] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:22 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:22.446145 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bab97e5-a466-4d30-bfa1-436fc23a8bc9 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:22.448219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-9bab97e5-a466-4d30-bfa1-436fc23a8bc9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:22.567863 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bab97e5-a466-4d30-bfa1-436fc23a8bc9 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:22.579476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f4018c-9e4a-4169-a47c-8625ab51dedd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:22.580106 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f4018c-9e4a-4169-a47c-8625ab51dedd service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:22.580389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f4018c-9e4a-4169-a47c-8625ab51dedd service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:22.580680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f4018c-9e4a-4169-a47c-8625ab51dedd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:22.598698 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f4018c-9e4a-4169-a47c-8625ab51dedd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:22.600213 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f4018c-9e4a-4169-a47c-8625ab51dedd service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:22.625650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1209/2407] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:48:22 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:48:27.058366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0c0479-cfff-4862-9172-c49f5651e013 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:27.059520 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0c0479-cfff-4862-9172-c49f5651e013 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:27.059868 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0c0479-cfff-4862-9172-c49f5651e013 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:27.087540 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-fd0c0479-cfff-4862-9172-c49f5651e013 None None] MFA Rules not processed for user `4d46ec34caf74ea7b7c8f836c2f7ea6c`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:27.088986 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fd0c0479-cfff-4862-9172-c49f5651e013 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:27.151055 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1199/2408] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:27 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:27.448734 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f4018c-9e4a-4169-a47c-8625ab51dedd service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:27.542626 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f4018c-9e4a-4169-a47c-8625ab51dedd service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:27.551221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a5cf60-d7ae-4c5c-9635-a1f3ba6a7f81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:27.551902 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a5cf60-d7ae-4c5c-9635-a1f3ba6a7f81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:27.552180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a5cf60-d7ae-4c5c-9635-a1f3ba6a7f81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:27.552446 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a5cf60-d7ae-4c5c-9635-a1f3ba6a7f81 admin admin] PATH_INFO: `/v3/users/44043fefdb034ee7adef616458c4ca36` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:27.553561 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a5cf60-d7ae-4c5c-9635-a1f3ba6a7f81 admin admin] RBAC: Authorizing `identity:delete_user(user_id=44043fefdb034ee7adef616458c4ca36)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:27.555757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a5cf60-d7ae-4c5c-9635-a1f3ba6a7f81 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:27.801155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-59a5cf60-d7ae-4c5c-9635-a1f3ba6a7f81 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '44043fefdb034ee7adef616458c4ca36'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:27.822116 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-59a5cf60-d7ae-4c5c-9635-a1f3ba6a7f81 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '44043fefdb034ee7adef616458c4ca36'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:27.822907 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-59a5cf60-d7ae-4c5c-9635-a1f3ba6a7f81 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '44043fefdb034ee7adef616458c4ca36'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:27.951686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-59a5cf60-d7ae-4c5c-9635-a1f3ba6a7f81 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '44043fefdb034ee7adef616458c4ca36'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:27.959644 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-59a5cf60-d7ae-4c5c-9635-a1f3ba6a7f81 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '44043fefdb034ee7adef616458c4ca36'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:27.969194 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1210/2409] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:48:27 2022] DELETE /identity/v3/users/44043fefdb034ee7adef616458c4ca36 => generated 0 bytes in 518 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:27.977585 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cda02a-3f1d-4c0a-9a44-d2e18fefd2f7 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:27.977956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cda02a-3f1d-4c0a-9a44-d2e18fefd2f7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:27.978450 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cda02a-3f1d-4c0a-9a44-d2e18fefd2f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:28.007280 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-69cda02a-3f1d-4c0a-9a44-d2e18fefd2f7 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:28.008608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-69cda02a-3f1d-4c0a-9a44-d2e18fefd2f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:28.023578 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:28.112095 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1200/2410] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:28.126204 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a5cf60-d7ae-4c5c-9635-a1f3ba6a7f81 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:28.127798 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-59a5cf60-d7ae-4c5c-9635-a1f3ba6a7f81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:28.151074 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:28.217517 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a5cf60-d7ae-4c5c-9635-a1f3ba6a7f81 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:28.222977 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a898d4-e4a6-444d-ba35-24446cfa87aa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:28.223858 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a898d4-e4a6-444d-ba35-24446cfa87aa service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:28.223858 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a898d4-e4a6-444d-ba35-24446cfa87aa service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:28.224104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a898d4-e4a6-444d-ba35-24446cfa87aa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:28.234941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a898d4-e4a6-444d-ba35-24446cfa87aa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:28.236357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a898d4-e4a6-444d-ba35-24446cfa87aa service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:28.248696 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1211/2411] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:48:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:48:29.033540 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69cda02a-3f1d-4c0a-9a44-d2e18fefd2f7 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:29.035499 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-69cda02a-3f1d-4c0a-9a44-d2e18fefd2f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:29.055686 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69cda02a-3f1d-4c0a-9a44-d2e18fefd2f7 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:29.060717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b605bf-f9ee-4880-8c7f-44b85e43d256 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:29.061296 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b605bf-f9ee-4880-8c7f-44b85e43d256 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:29.063702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b605bf-f9ee-4880-8c7f-44b85e43d256 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:29.063702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b605bf-f9ee-4880-8c7f-44b85e43d256 admin admin] PATH_INFO: `/v3/projects/7e897f8836fe4715ab1b8ec5abf574d7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:29.063702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b605bf-f9ee-4880-8c7f-44b85e43d256 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7e897f8836fe4715ab1b8ec5abf574d7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:29.065092 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b605bf-f9ee-4880-8c7f-44b85e43d256 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:29.217538 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c1b605bf-f9ee-4880-8c7f-44b85e43d256 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7e897f8836fe4715ab1b8ec5abf574d7', 'request_id': 'req-c1b605bf-f9ee-4880-8c7f-44b85e43d256'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:29.228729 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c1b605bf-f9ee-4880-8c7f-44b85e43d256 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7e897f8836fe4715ab1b8ec5abf574d7', 'request_id': 'req-c1b605bf-f9ee-4880-8c7f-44b85e43d256'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:29.239372 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c1b605bf-f9ee-4880-8c7f-44b85e43d256 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7e897f8836fe4715ab1b8ec5abf574d7', 'request_id': 'req-c1b605bf-f9ee-4880-8c7f-44b85e43d256'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:29.302925 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a937730b-1378-4862-9847-ef942046bcbe None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:29.303521 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a937730b-1378-4862-9847-ef942046bcbe None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:29.303719 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a937730b-1378-4862-9847-ef942046bcbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:29.327045 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-a937730b-1378-4862-9847-ef942046bcbe None None] MFA Rules not processed for user `8c4ced79c3ff4842b30b472acb555cd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:29.328570 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a937730b-1378-4862-9847-ef942046bcbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:29.394690 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c1b605bf-f9ee-4880-8c7f-44b85e43d256 admin admin] The token cache is being invalidate because project 7e897f8836fe4715ab1b8ec5abf574d7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:48:29.394984 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c1b605bf-f9ee-4880-8c7f-44b85e43d256 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:29.400494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c1b605bf-f9ee-4880-8c7f-44b85e43d256 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:29.400494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1201/2412] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:48:29 2022] DELETE /identity/v3/projects/7e897f8836fe4715ab1b8ec5abf574d7 => generated 0 bytes in 364 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:29.404893 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5268199-81da-40a7-a38a-d5c22ec9f187 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:29.405356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5268199-81da-40a7-a38a-d5c22ec9f187 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:29.405830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5268199-81da-40a7-a38a-d5c22ec9f187 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:29.406900 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1212/2413] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:29 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:29.433352 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c5268199-81da-40a7-a38a-d5c22ec9f187 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:29.434769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c5268199-81da-40a7-a38a-d5c22ec9f187 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:29.539434 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1202/2414] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:29.548431 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5104ba-0b85-4a10-b0cf-f1002ebd4993 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:29.548780 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5104ba-0b85-4a10-b0cf-f1002ebd4993 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:29.549121 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5104ba-0b85-4a10-b0cf-f1002ebd4993 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:29.573063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-da5104ba-0b85-4a10-b0cf-f1002ebd4993 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:29.574886 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-da5104ba-0b85-4a10-b0cf-f1002ebd4993 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:29.583601 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1213/2415] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:29.588240 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5268199-81da-40a7-a38a-d5c22ec9f187 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:29.600127 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5268199-81da-40a7-a38a-d5c22ec9f187 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:29.605716 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083cf123-a249-4645-810a-355f0091253c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:29.606644 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083cf123-a249-4645-810a-355f0091253c admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:29.607279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083cf123-a249-4645-810a-355f0091253c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:29.607557 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083cf123-a249-4645-810a-355f0091253c admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:29.608523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083cf123-a249-4645-810a-355f0091253c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:48:29.609166 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083cf123-a249-4645-810a-355f0091253c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:29.618766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083cf123-a249-4645-810a-355f0091253c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:29.635197 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1203/2416] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:48:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:48:29.640360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5104ba-0b85-4a10-b0cf-f1002ebd4993 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:29.650658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5104ba-0b85-4a10-b0cf-f1002ebd4993 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:29.659532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b216e7e4-5803-4e75-8cc1-986ee7f30d93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:29.660454 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b216e7e4-5803-4e75-8cc1-986ee7f30d93 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:29.660825 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b216e7e4-5803-4e75-8cc1-986ee7f30d93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:29.661099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b216e7e4-5803-4e75-8cc1-986ee7f30d93 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:29.661767 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b216e7e4-5803-4e75-8cc1-986ee7f30d93 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:29.663987 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b216e7e4-5803-4e75-8cc1-986ee7f30d93 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:29.770915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1214/2417] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:48:29 2022] POST /identity/v3/projects => generated 373 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:48:29.776230 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083cf123-a249-4645-810a-355f0091253c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:29.790850 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083cf123-a249-4645-810a-355f0091253c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:29.821493 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:29.867787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe6d152-aad9-4335-a441-7caf373a54eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:29.868870 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe6d152-aad9-4335-a441-7caf373a54eb admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:29.868870 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe6d152-aad9-4335-a441-7caf373a54eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:29.869106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe6d152-aad9-4335-a441-7caf373a54eb admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:29.869568 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe6d152-aad9-4335-a441-7caf373a54eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:29.872118 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe6d152-aad9-4335-a441-7caf373a54eb admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:29.919100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b62c78-b18c-4c4b-b31b-ce0dc78f3a11 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:29.919499 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b62c78-b18c-4c4b-b31b-ce0dc78f3a11 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:29.919680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b62c78-b18c-4c4b-b31b-ce0dc78f3a11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:29.940684 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-e1b62c78-b18c-4c4b-b31b-ce0dc78f3a11 None None] MFA Rules not processed for user `7ad21da05fc246d489b0afa641f20216`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:29.941951 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-e1b62c78-b18c-4c4b-b31b-ce0dc78f3a11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:29.969538 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1204/2418] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:48:29 2022] POST /identity/v3/users => generated 287 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:48:29.973777 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe6d152-aad9-4335-a441-7caf373a54eb admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:29.986025 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe6d152-aad9-4335-a441-7caf373a54eb admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:29.991174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0e79b1-9840-4e4d-8042-e4214e249340 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:29.991881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0e79b1-9840-4e4d-8042-e4214e249340 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:29.993143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0e79b1-9840-4e4d-8042-e4214e249340 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:29.993495 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0e79b1-9840-4e4d-8042-e4214e249340 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:29.994225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0e79b1-9840-4e4d-8042-e4214e249340 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:29.996586 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0e79b1-9840-4e4d-8042-e4214e249340 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:30.008548 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1205/2419] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:29 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:48:30.013132 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0e79b1-9840-4e4d-8042-e4214e249340 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:30.014627 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1215/2420] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:29 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:48:30.024439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0e79b1-9840-4e4d-8042-e4214e249340 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:30.029709 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f17f0fa-ec2d-4640-923b-0ae434c1a76f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:30.030514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f17f0fa-ec2d-4640-923b-0ae434c1a76f admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:30.032997 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f17f0fa-ec2d-4640-923b-0ae434c1a76f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:30.032997 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f17f0fa-ec2d-4640-923b-0ae434c1a76f admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:30.032997 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f17f0fa-ec2d-4640-923b-0ae434c1a76f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:30.033715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f17f0fa-ec2d-4640-923b-0ae434c1a76f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:30.049112 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1206/2421] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:30 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:48:30.050314 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b62c78-b18c-4c4b-b31b-ce0dc78f3a11 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:30.063489 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b62c78-b18c-4c4b-b31b-ce0dc78f3a11 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:30.069583 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e911df-836b-404c-adb6-ecb27b374dc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:30.070260 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e911df-836b-404c-adb6-ecb27b374dc6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:30.070540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e911df-836b-404c-adb6-ecb27b374dc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:30.070893 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e911df-836b-404c-adb6-ecb27b374dc6 admin admin] PATH_INFO: `/v3/projects/1330031e214045bf9d45c73da1445255/users/20bee9f38eb14541a393037eaa389a44/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:30.089440 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e911df-836b-404c-adb6-ecb27b374dc6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1330031e214045bf9d45c73da1445255, user_id=20bee9f38eb14541a393037eaa389a44, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:30.092497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e911df-836b-404c-adb6-ecb27b374dc6 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:30.210196 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1216/2422] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:48:30 2022] PUT /identity/v3/projects/1330031e214045bf9d45c73da1445255/users/20bee9f38eb14541a393037eaa389a44/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:48:30.215838 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2166bedf-9a27-4731-88c1-4f451ba7f9e6 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:30.216223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2166bedf-9a27-4731-88c1-4f451ba7f9e6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:30.216596 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2166bedf-9a27-4731-88c1-4f451ba7f9e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:30.240439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-2166bedf-9a27-4731-88c1-4f451ba7f9e6 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:30.240439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2166bedf-9a27-4731-88c1-4f451ba7f9e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:30.339760 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1207/2423] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:30.349665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e911df-836b-404c-adb6-ecb27b374dc6 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:30.351628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-12e911df-836b-404c-adb6-ecb27b374dc6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:30.381598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:30.425616 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e911df-836b-404c-adb6-ecb27b374dc6 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:30.429722 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1c8922-7f2a-4c5b-92c6-6d6ed110c6d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:30.431565 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1c8922-7f2a-4c5b-92c6-6d6ed110c6d6 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:30.431923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1c8922-7f2a-4c5b-92c6-6d6ed110c6d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:30.432268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1c8922-7f2a-4c5b-92c6-6d6ed110c6d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:30.442664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1c8922-7f2a-4c5b-92c6-6d6ed110c6d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:30.444134 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1c8922-7f2a-4c5b-92c6-6d6ed110c6d6 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:30.455236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1217/2424] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:48:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:48:31.014570 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2166bedf-9a27-4731-88c1-4f451ba7f9e6 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:31.016337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2166bedf-9a27-4731-88c1-4f451ba7f9e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:31.035107 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2166bedf-9a27-4731-88c1-4f451ba7f9e6 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:31.043931 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4d26c2-236d-42ee-99d5-41197602570e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:31.047949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4d26c2-236d-42ee-99d5-41197602570e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:31.047949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4d26c2-236d-42ee-99d5-41197602570e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:31.047949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4d26c2-236d-42ee-99d5-41197602570e admin admin] PATH_INFO: `/v3/users/135fe9bb598d4a489ceea853be586c03` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:31.047949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4d26c2-236d-42ee-99d5-41197602570e admin admin] RBAC: Authorizing `identity:delete_user(user_id=135fe9bb598d4a489ceea853be586c03)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:31.049961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4d26c2-236d-42ee-99d5-41197602570e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:31.323805 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ec4d26c2-236d-42ee-99d5-41197602570e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '135fe9bb598d4a489ceea853be586c03'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:31.325010 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ec4d26c2-236d-42ee-99d5-41197602570e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '135fe9bb598d4a489ceea853be586c03'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:31.325723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ec4d26c2-236d-42ee-99d5-41197602570e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '135fe9bb598d4a489ceea853be586c03'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:31.343097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ec4d26c2-236d-42ee-99d5-41197602570e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '135fe9bb598d4a489ceea853be586c03'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:31.436766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ec4d26c2-236d-42ee-99d5-41197602570e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '135fe9bb598d4a489ceea853be586c03'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:31.461436 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1208/2425] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:48:31 2022] DELETE /identity/v3/users/135fe9bb598d4a489ceea853be586c03 => generated 0 bytes in 447 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:31.472018 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0265f4fd-d720-4a10-bfff-c44e9f4ffdec None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:31.472598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0265f4fd-d720-4a10-bfff-c44e9f4ffdec None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:31.472904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0265f4fd-d720-4a10-bfff-c44e9f4ffdec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:31.496476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-0265f4fd-d720-4a10-bfff-c44e9f4ffdec None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:31.497867 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0265f4fd-d720-4a10-bfff-c44e9f4ffdec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:31.566449 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1218/2426] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:31.587235 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4d26c2-236d-42ee-99d5-41197602570e admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:31.588799 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ec4d26c2-236d-42ee-99d5-41197602570e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:31.600835 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:31.690746 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4d26c2-236d-42ee-99d5-41197602570e admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:31.712123 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99622c83-dc08-4af8-9967-2374dc3eaf81 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:31.712815 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99622c83-dc08-4af8-9967-2374dc3eaf81 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:31.713192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99622c83-dc08-4af8-9967-2374dc3eaf81 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:31.713773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99622c83-dc08-4af8-9967-2374dc3eaf81 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:31.760152 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99622c83-dc08-4af8-9967-2374dc3eaf81 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:31.762164 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99622c83-dc08-4af8-9967-2374dc3eaf81 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:31.785688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1209/2427] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:48:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:48:32.295491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0265f4fd-d720-4a10-bfff-c44e9f4ffdec None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:32.299187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0265f4fd-d720-4a10-bfff-c44e9f4ffdec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:32.322151 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0265f4fd-d720-4a10-bfff-c44e9f4ffdec None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:32.343025 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eec51b3-2e43-4f8d-a2ba-fdc17534cc73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:32.344292 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eec51b3-2e43-4f8d-a2ba-fdc17534cc73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:32.346327 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eec51b3-2e43-4f8d-a2ba-fdc17534cc73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:32.347307 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eec51b3-2e43-4f8d-a2ba-fdc17534cc73 admin admin] PATH_INFO: `/v3/projects/3aced3101122427ebab228330526aaca` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:32.355904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eec51b3-2e43-4f8d-a2ba-fdc17534cc73 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3aced3101122427ebab228330526aaca)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:32.360274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eec51b3-2e43-4f8d-a2ba-fdc17534cc73 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:32.543202 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6eec51b3-2e43-4f8d-a2ba-fdc17534cc73 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3aced3101122427ebab228330526aaca', 'request_id': 'req-6eec51b3-2e43-4f8d-a2ba-fdc17534cc73'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:32.552849 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6eec51b3-2e43-4f8d-a2ba-fdc17534cc73 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3aced3101122427ebab228330526aaca', 'request_id': 'req-6eec51b3-2e43-4f8d-a2ba-fdc17534cc73'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:32.574299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6eec51b3-2e43-4f8d-a2ba-fdc17534cc73 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3aced3101122427ebab228330526aaca', 'request_id': 'req-6eec51b3-2e43-4f8d-a2ba-fdc17534cc73'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:32.710932 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6eec51b3-2e43-4f8d-a2ba-fdc17534cc73 admin admin] The token cache is being invalidate because project 3aced3101122427ebab228330526aaca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:48:32.711521 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6eec51b3-2e43-4f8d-a2ba-fdc17534cc73 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:32.712828 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6eec51b3-2e43-4f8d-a2ba-fdc17534cc73 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:32.715766 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1219/2428] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:48:32 2022] DELETE /identity/v3/projects/3aced3101122427ebab228330526aaca => generated 0 bytes in 421 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:32.721024 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cddafe6-a503-4634-b24a-bedb132b7577 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:32.722047 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cddafe6-a503-4634-b24a-bedb132b7577 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:32.723114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cddafe6-a503-4634-b24a-bedb132b7577 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:32.758938 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-6cddafe6-a503-4634-b24a-bedb132b7577 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:32.761586 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6cddafe6-a503-4634-b24a-bedb132b7577 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:32.905919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1210/2429] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:32.915271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43dfbfc-d43f-4627-be7f-7368efe24c82 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:32.916911 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43dfbfc-d43f-4627-be7f-7368efe24c82 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:32.917401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43dfbfc-d43f-4627-be7f-7368efe24c82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:32.947774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-c43dfbfc-d43f-4627-be7f-7368efe24c82 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:32.950552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c43dfbfc-d43f-4627-be7f-7368efe24c82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:32.959615 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1220/2430] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:32.964353 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cddafe6-a503-4634-b24a-bedb132b7577 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:32.974836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cddafe6-a503-4634-b24a-bedb132b7577 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:32.979508 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e7be2b-685a-46d9-8275-41659dfa18ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:32.980809 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e7be2b-685a-46d9-8275-41659dfa18ba admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:32.981173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e7be2b-685a-46d9-8275-41659dfa18ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:32.981507 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e7be2b-685a-46d9-8275-41659dfa18ba admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:32.982213 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e7be2b-685a-46d9-8275-41659dfa18ba admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:48:32.982577 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e7be2b-685a-46d9-8275-41659dfa18ba admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:32.984255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e7be2b-685a-46d9-8275-41659dfa18ba admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:33.004233 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1211/2431] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:48:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:48:33.008473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43dfbfc-d43f-4627-be7f-7368efe24c82 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:33.029084 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43dfbfc-d43f-4627-be7f-7368efe24c82 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:33.035082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5787e4d2-a05f-4c3b-9815-d4dac7af62b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:33.036152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5787e4d2-a05f-4c3b-9815-d4dac7af62b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:33.036764 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5787e4d2-a05f-4c3b-9815-d4dac7af62b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:33.037253 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5787e4d2-a05f-4c3b-9815-d4dac7af62b9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:33.038097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5787e4d2-a05f-4c3b-9815-d4dac7af62b9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:33.040480 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5787e4d2-a05f-4c3b-9815-d4dac7af62b9 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:33.138673 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1221/2432] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:48:33 2022] POST /identity/v3/projects => generated 375 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:48:33.143792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e7be2b-685a-46d9-8275-41659dfa18ba admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:33.167198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e7be2b-685a-46d9-8275-41659dfa18ba admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:33.235052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00834eb2-34cb-4ba1-894e-7e1de3a8c184 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:33.235769 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00834eb2-34cb-4ba1-894e-7e1de3a8c184 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:33.236049 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00834eb2-34cb-4ba1-894e-7e1de3a8c184 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:33.236371 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00834eb2-34cb-4ba1-894e-7e1de3a8c184 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:33.237019 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00834eb2-34cb-4ba1-894e-7e1de3a8c184 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:33.239463 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00834eb2-34cb-4ba1-894e-7e1de3a8c184 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:33.319314 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3ebff9-fdd4-496e-bf7a-7461f504862a None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:33.319685 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3ebff9-fdd4-496e-bf7a-7461f504862a None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:33.319882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3ebff9-fdd4-496e-bf7a-7461f504862a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:33.347274 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-9b3ebff9-fdd4-496e-bf7a-7461f504862a None None] MFA Rules not processed for user `7d6e0545a40a436d8c63820b795551a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:33.348542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-9b3ebff9-fdd4-496e-bf7a-7461f504862a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:33.356732 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:33.402978 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1212/2433] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:48:33 2022] POST /identity/v3/users => generated 288 bytes in 260 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:48:33.407050 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00834eb2-34cb-4ba1-894e-7e1de3a8c184 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:33.426976 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00834eb2-34cb-4ba1-894e-7e1de3a8c184 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:33.432626 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b1f181-1c5d-49aa-8230-a726abac96c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:33.434009 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b1f181-1c5d-49aa-8230-a726abac96c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:33.434440 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b1f181-1c5d-49aa-8230-a726abac96c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:33.434920 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b1f181-1c5d-49aa-8230-a726abac96c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:33.435784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b1f181-1c5d-49aa-8230-a726abac96c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:33.437656 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b1f181-1c5d-49aa-8230-a726abac96c5 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:33.449820 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1213/2434] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:33 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:48:33.459778 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b1f181-1c5d-49aa-8230-a726abac96c5 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:33.471893 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b1f181-1c5d-49aa-8230-a726abac96c5 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:33.477407 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3a3c98-4bbf-4b76-9f48-dbeee01914c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:33.478633 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3a3c98-4bbf-4b76-9f48-dbeee01914c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:33.479066 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3a3c98-4bbf-4b76-9f48-dbeee01914c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:33.479520 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3a3c98-4bbf-4b76-9f48-dbeee01914c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:33.480370 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e3a3c98-4bbf-4b76-9f48-dbeee01914c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:33.482128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e3a3c98-4bbf-4b76-9f48-dbeee01914c5 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:33.485533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1222/2435] 10.222.0.39 () {58 vars in 1009 bytes} [Fri Sep 30 09:48:33 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:48:33.495244 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1214/2436] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:33 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:48:33.500485 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3ebff9-fdd4-496e-bf7a-7461f504862a None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:33.502135 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-9b3ebff9-fdd4-496e-bf7a-7461f504862a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:33.508624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3a3c98-4bbf-4b76-9f48-dbeee01914c5 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:33.525049 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:33.541285 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3a3c98-4bbf-4b76-9f48-dbeee01914c5 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:33.568831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288bd95e-67a8-41d3-9222-1780806b56ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:33.568831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288bd95e-67a8-41d3-9222-1780806b56ed admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:33.568831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288bd95e-67a8-41d3-9222-1780806b56ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:33.568831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288bd95e-67a8-41d3-9222-1780806b56ed admin admin] PATH_INFO: `/v3/projects/068a17bb811448d28e4ba1907f47b8a1/users/6ebe000d3eb147aca44db7f213183c14/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:33.642511 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288bd95e-67a8-41d3-9222-1780806b56ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=068a17bb811448d28e4ba1907f47b8a1, user_id=6ebe000d3eb147aca44db7f213183c14, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:33.660636 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288bd95e-67a8-41d3-9222-1780806b56ed admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:33.731251 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3ebff9-fdd4-496e-bf7a-7461f504862a None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:33.748552 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1215/2437] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:48:33 2022] PUT /identity/v3/projects/068a17bb811448d28e4ba1907f47b8a1/users/6ebe000d3eb147aca44db7f213183c14/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:33.772350 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a25d3f6-8c17-43a8-b17d-a22eabff9a50 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:33.773531 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a25d3f6-8c17-43a8-b17d-a22eabff9a50 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:33.776723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a25d3f6-8c17-43a8-b17d-a22eabff9a50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:33.781658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49298fbd-f3cb-49ca-b736-95f622d494fa service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e5ad921e6fa41c297696ec7279f0fc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:33.782909 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49298fbd-f3cb-49ca-b736-95f622d494fa service swift] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:33.784223 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49298fbd-f3cb-49ca-b736-95f622d494fa service swift] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:33.787153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49298fbd-f3cb-49ca-b736-95f622d494fa service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:33.811376 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49298fbd-f3cb-49ca-b736-95f622d494fa service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:33.815618 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49298fbd-f3cb-49ca-b736-95f622d494fa service swift] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:33.821914 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9a25d3f6-8c17-43a8-b17d-a22eabff9a50 None None] MFA Rules not processed for user `6ebe000d3eb147aca44db7f213183c14`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:33.827151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9a25d3f6-8c17-43a8-b17d-a22eabff9a50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:33.838209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1223/2438] 10.222.0.39 () {60 vars in 1384 bytes} [Fri Sep 30 09:48:33 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 30 09:48:33.976738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1216/2439] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:33 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:33.988055 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49298fbd-f3cb-49ca-b736-95f622d494fa service swift] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:33.989737 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-49298fbd-f3cb-49ca-b736-95f622d494fa service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:34.233968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49298fbd-f3cb-49ca-b736-95f622d494fa service swift] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:34.429929 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dd4a79-56e2-4a8f-abf6-29a262573fd0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:34.431082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dd4a79-56e2-4a8f-abf6-29a262573fd0 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:34.432351 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dd4a79-56e2-4a8f-abf6-29a262573fd0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:34.432705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dd4a79-56e2-4a8f-abf6-29a262573fd0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:34.454087 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dd4a79-56e2-4a8f-abf6-29a262573fd0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:34.455438 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dd4a79-56e2-4a8f-abf6-29a262573fd0 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:34.483435 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1224/2440] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:48:33 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 496 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:48:35.463873 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb88398-3bdb-4516-9b87-dbc93dbfa593 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:35.464523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb88398-3bdb-4516-9b87-dbc93dbfa593 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:35.464523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb88398-3bdb-4516-9b87-dbc93dbfa593 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:35.527690 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ecb88398-3bdb-4516-9b87-dbc93dbfa593 None None] MFA Rules not processed for user `587169e32cf64fe5b684e4b2ce0701a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:35.532290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ecb88398-3bdb-4516-9b87-dbc93dbfa593 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:35.760108 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1217/2441] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:35 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 294 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:48:37.217706 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77491d25-6a46-4ab0-9493-6471f01f337c None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:37.218320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77491d25-6a46-4ab0-9493-6471f01f337c None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:37.218320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77491d25-6a46-4ab0-9493-6471f01f337c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:37.237393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:37.274004 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-77491d25-6a46-4ab0-9493-6471f01f337c None None] MFA Rules not processed for user `6ebe000d3eb147aca44db7f213183c14`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:37.275369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-77491d25-6a46-4ab0-9493-6471f01f337c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:37.312236 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1225/2442] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:37 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:48:41.546227 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2172ea-534f-4f70-8414-3fc6c2af93b6 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:41.547176 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2172ea-534f-4f70-8414-3fc6c2af93b6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:41.547653 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2172ea-534f-4f70-8414-3fc6c2af93b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:41.613019 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ad2172ea-534f-4f70-8414-3fc6c2af93b6 None None] MFA Rules not processed for user `20bee9f38eb14541a393037eaa389a44`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:41.615211 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ad2172ea-534f-4f70-8414-3fc6c2af93b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:41.827723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1218/2443] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:41 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 283 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:41.832492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77491d25-6a46-4ab0-9493-6471f01f337c None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:41.835023 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-77491d25-6a46-4ab0-9493-6471f01f337c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:42.020496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77491d25-6a46-4ab0-9493-6471f01f337c None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:42.026333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257eb3a6-d554-41cc-a8e7-55fd57806ce1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:42.030074 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257eb3a6-d554-41cc-a8e7-55fd57806ce1 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:42.030074 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257eb3a6-d554-41cc-a8e7-55fd57806ce1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:42.030074 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257eb3a6-d554-41cc-a8e7-55fd57806ce1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:42.030074 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257eb3a6-d554-41cc-a8e7-55fd57806ce1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:42.033200 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257eb3a6-d554-41cc-a8e7-55fd57806ce1 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:42.176346 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1226/2444] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:48:41 2022] POST /identity/v3/projects => generated 373 bytes in 344 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:48:42.180791 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2172ea-534f-4f70-8414-3fc6c2af93b6 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:42.192555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:42.206249 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2172ea-534f-4f70-8414-3fc6c2af93b6 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:42.324532 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73059fc4-3c9f-4a45-9775-5e1d69150c96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:42.326814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73059fc4-3c9f-4a45-9775-5e1d69150c96 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:42.327184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73059fc4-3c9f-4a45-9775-5e1d69150c96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:42.327538 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73059fc4-3c9f-4a45-9775-5e1d69150c96 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:42.328277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73059fc4-3c9f-4a45-9775-5e1d69150c96 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:42.330327 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73059fc4-3c9f-4a45-9775-5e1d69150c96 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:42.450741 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1219/2445] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:48:42 2022] POST /identity/v3/users => generated 293 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:48:42.454932 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257eb3a6-d554-41cc-a8e7-55fd57806ce1 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:42.471223 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257eb3a6-d554-41cc-a8e7-55fd57806ce1 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:42.488466 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6779c6-3018-4cc1-aed0-60c7812b7e3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:42.489788 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6779c6-3018-4cc1-aed0-60c7812b7e3d admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:42.490231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6779c6-3018-4cc1-aed0-60c7812b7e3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:42.491285 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6779c6-3018-4cc1-aed0-60c7812b7e3d admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:42.492628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6779c6-3018-4cc1-aed0-60c7812b7e3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:42.500362 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6779c6-3018-4cc1-aed0-60c7812b7e3d admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:42.552391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1227/2446] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:42 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:48:42.556705 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73059fc4-3c9f-4a45-9775-5e1d69150c96 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:42.580083 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73059fc4-3c9f-4a45-9775-5e1d69150c96 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:42.591461 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2db5a61-6b36-4b44-af8f-2dee29f54f25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:42.596960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2db5a61-6b36-4b44-af8f-2dee29f54f25 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:42.597471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2db5a61-6b36-4b44-af8f-2dee29f54f25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:42.598005 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2db5a61-6b36-4b44-af8f-2dee29f54f25 admin admin] PATH_INFO: `/v3/projects/82b48c489465430a836de26d4aa39440/users/29a0903dfdb042b8aea469db4fe31599/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:42.626806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2db5a61-6b36-4b44-af8f-2dee29f54f25 admin admin] RBAC: Authorizing `identity:create_grant(project_id=82b48c489465430a836de26d4aa39440, user_id=29a0903dfdb042b8aea469db4fe31599, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:42.631603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2db5a61-6b36-4b44-af8f-2dee29f54f25 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:42.742738 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6779c6-3018-4cc1-aed0-60c7812b7e3d admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:42.743242 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1220/2447] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:48:42 2022] PUT /identity/v3/projects/82b48c489465430a836de26d4aa39440/users/29a0903dfdb042b8aea469db4fe31599/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:48:42.759210 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6779c6-3018-4cc1-aed0-60c7812b7e3d admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:42.926483 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df83d56-685b-4736-929d-542f11da2ef7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:42.927389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df83d56-685b-4736-929d-542f11da2ef7 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:42.927635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df83d56-685b-4736-929d-542f11da2ef7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:42.927926 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df83d56-685b-4736-929d-542f11da2ef7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:42.928676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df83d56-685b-4736-929d-542f11da2ef7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:42.933216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df83d56-685b-4736-929d-542f11da2ef7 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:42.960473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1228/2448] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:42 2022] GET /identity/v3/roles => generated 1420 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:48:42.964334 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2db5a61-6b36-4b44-af8f-2dee29f54f25 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:42.984223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2db5a61-6b36-4b44-af8f-2dee29f54f25 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:43.000656 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08355462-eeaa-4b59-8112-862aa8f1a88e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:43.001435 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08355462-eeaa-4b59-8112-862aa8f1a88e admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:43.001724 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08355462-eeaa-4b59-8112-862aa8f1a88e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:43.002215 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08355462-eeaa-4b59-8112-862aa8f1a88e admin admin] PATH_INFO: `/v3/projects/82b48c489465430a836de26d4aa39440/users/29a0903dfdb042b8aea469db4fe31599/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:43.009899 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08355462-eeaa-4b59-8112-862aa8f1a88e admin admin] RBAC: Authorizing `identity:create_grant(project_id=82b48c489465430a836de26d4aa39440, user_id=29a0903dfdb042b8aea469db4fe31599, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:43.015684 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08355462-eeaa-4b59-8112-862aa8f1a88e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:43.046918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1221/2449] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:48:42 2022] PUT /identity/v3/projects/82b48c489465430a836de26d4aa39440/users/29a0903dfdb042b8aea469db4fe31599/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:51.462056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df83d56-685b-4736-929d-542f11da2ef7 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:51.465222 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7df83d56-685b-4736-929d-542f11da2ef7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:51.483944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df83d56-685b-4736-929d-542f11da2ef7 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:51.558180 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eedb172a-0512-4e88-b072-0d46a011c25e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:51.559253 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedb172a-0512-4e88-b072-0d46a011c25e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:51.559775 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedb172a-0512-4e88-b072-0d46a011c25e admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:51.560295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedb172a-0512-4e88-b072-0d46a011c25e admin admin] PATH_INFO: `/v3/users/1c1e525815e9407a8f5d8c58b0b7d6fa` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:51.561872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eedb172a-0512-4e88-b072-0d46a011c25e admin admin] RBAC: Authorizing `identity:delete_user(user_id=1c1e525815e9407a8f5d8c58b0b7d6fa)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:51.563965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eedb172a-0512-4e88-b072-0d46a011c25e admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:51.579833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:51.892186 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-eedb172a-0512-4e88-b072-0d46a011c25e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1c1e525815e9407a8f5d8c58b0b7d6fa'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:51.914807 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-eedb172a-0512-4e88-b072-0d46a011c25e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1c1e525815e9407a8f5d8c58b0b7d6fa'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:51.917349 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-eedb172a-0512-4e88-b072-0d46a011c25e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1c1e525815e9407a8f5d8c58b0b7d6fa'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:51.984258 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-eedb172a-0512-4e88-b072-0d46a011c25e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1c1e525815e9407a8f5d8c58b0b7d6fa'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:52.002234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-eedb172a-0512-4e88-b072-0d46a011c25e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1c1e525815e9407a8f5d8c58b0b7d6fa'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:52.018099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1229/2450] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:48:51 2022] DELETE /identity/v3/users/1c1e525815e9407a8f5d8c58b0b7d6fa => generated 0 bytes in 555 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:52.021358 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f27ce38-8e62-48bf-be9b-815fea5e458f None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:52.023371 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f27ce38-8e62-48bf-be9b-815fea5e458f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:52.023371 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f27ce38-8e62-48bf-be9b-815fea5e458f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:52.044794 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7f27ce38-8e62-48bf-be9b-815fea5e458f None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:52.046753 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7f27ce38-8e62-48bf-be9b-815fea5e458f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:52.152584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1222/2451] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:52.162367 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eedb172a-0512-4e88-b072-0d46a011c25e admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:52.165538 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-eedb172a-0512-4e88-b072-0d46a011c25e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:52.253469 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eedb172a-0512-4e88-b072-0d46a011c25e admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:52.257750 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11057db-21b7-4343-ab61-21d9508186a2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:52.259702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11057db-21b7-4343-ab61-21d9508186a2 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:52.259702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11057db-21b7-4343-ab61-21d9508186a2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:52.259702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11057db-21b7-4343-ab61-21d9508186a2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:52.273523 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11057db-21b7-4343-ab61-21d9508186a2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:52.275172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11057db-21b7-4343-ab61-21d9508186a2 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:52.290651 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1230/2452] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:48:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:48:53.560584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f27ce38-8e62-48bf-be9b-815fea5e458f None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:53.562101 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7f27ce38-8e62-48bf-be9b-815fea5e458f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:53.580021 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f27ce38-8e62-48bf-be9b-815fea5e458f None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:53.586791 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e83c6eee-9a9a-4b38-b49e-5e213c7ceafe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:53.601629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83c6eee-9a9a-4b38-b49e-5e213c7ceafe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:53.601629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83c6eee-9a9a-4b38-b49e-5e213c7ceafe admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:53.601629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83c6eee-9a9a-4b38-b49e-5e213c7ceafe admin admin] PATH_INFO: `/v3/projects/13a92def01b54713a3acf8d15726c7d6` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:53.602723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83c6eee-9a9a-4b38-b49e-5e213c7ceafe admin admin] RBAC: Authorizing `identity:delete_project(project_id=13a92def01b54713a3acf8d15726c7d6)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:53.623352 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83c6eee-9a9a-4b38-b49e-5e213c7ceafe admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:54.208718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e83c6eee-9a9a-4b38-b49e-5e213c7ceafe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '13a92def01b54713a3acf8d15726c7d6', 'request_id': 'req-e83c6eee-9a9a-4b38-b49e-5e213c7ceafe'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:54.208718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=91,keystone:INSERT=5,keystone:DELETE=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:54.208718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e83c6eee-9a9a-4b38-b49e-5e213c7ceafe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '13a92def01b54713a3acf8d15726c7d6', 'request_id': 'req-e83c6eee-9a9a-4b38-b49e-5e213c7ceafe'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:54.212838 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e83c6eee-9a9a-4b38-b49e-5e213c7ceafe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '13a92def01b54713a3acf8d15726c7d6', 'request_id': 'req-e83c6eee-9a9a-4b38-b49e-5e213c7ceafe'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:54.598960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e83c6eee-9a9a-4b38-b49e-5e213c7ceafe admin admin] The token cache is being invalidate because project 13a92def01b54713a3acf8d15726c7d6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:48:54.599633 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e83c6eee-9a9a-4b38-b49e-5e213c7ceafe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:54.599938 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e83c6eee-9a9a-4b38-b49e-5e213c7ceafe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:54.601504 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1223/2453] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:48:53 2022] DELETE /identity/v3/projects/13a92def01b54713a3acf8d15726c7d6 => generated 0 bytes in 1042 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:54.607217 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01754b0-3a1f-42c3-81df-bb3b268c969d None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:54.608202 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01754b0-3a1f-42c3-81df-bb3b268c969d None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:54.608785 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01754b0-3a1f-42c3-81df-bb3b268c969d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:54.644109 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-f01754b0-3a1f-42c3-81df-bb3b268c969d None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:54.647957 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-f01754b0-3a1f-42c3-81df-bb3b268c969d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:54.791168 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1231/2454] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:54.799323 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93ed04b-9114-4cbe-8fde-412d809f9972 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:54.803184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93ed04b-9114-4cbe-8fde-412d809f9972 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:54.803184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93ed04b-9114-4cbe-8fde-412d809f9972 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:54.815669 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd7a090-cc07-4432-8900-d6ed4b75d974 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:54.816304 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd7a090-cc07-4432-8900-d6ed4b75d974 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:54.816655 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd7a090-cc07-4432-8900-d6ed4b75d974 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:54.841511 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e93ed04b-9114-4cbe-8fde-412d809f9972 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:54.843254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e93ed04b-9114-4cbe-8fde-412d809f9972 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:54.853332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1224/2455] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:54.858540 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93ed04b-9114-4cbe-8fde-412d809f9972 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:54.873255 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-bfd7a090-cc07-4432-8900-d6ed4b75d974 None None] MFA Rules not processed for user `29a0903dfdb042b8aea469db4fe31599`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:54.876956 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-bfd7a090-cc07-4432-8900-d6ed4b75d974 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:54.893089 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93ed04b-9114-4cbe-8fde-412d809f9972 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:54.902514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a253277-de0c-410b-8853-ed48c1bad4fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:54.903513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a253277-de0c-410b-8853-ed48c1bad4fa admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:54.903953 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a253277-de0c-410b-8853-ed48c1bad4fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:54.904494 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:54.905591 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a253277-de0c-410b-8853-ed48c1bad4fa admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:54.906315 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a253277-de0c-410b-8853-ed48c1bad4fa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:48:54.906837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a253277-de0c-410b-8853-ed48c1bad4fa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:54.908957 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a253277-de0c-410b-8853-ed48c1bad4fa admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:54.923013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1225/2456] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:48:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:48:54.926982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a253277-de0c-410b-8853-ed48c1bad4fa admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:54.940392 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a253277-de0c-410b-8853-ed48c1bad4fa admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:54.953031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b148f365-4484-4434-92b6-d692f84415d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:54.953840 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b148f365-4484-4434-92b6-d692f84415d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:54.954635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b148f365-4484-4434-92b6-d692f84415d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:54.956524 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b148f365-4484-4434-92b6-d692f84415d8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:54.959273 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b148f365-4484-4434-92b6-d692f84415d8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:54.963823 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b148f365-4484-4434-92b6-d692f84415d8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:55.000573 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1232/2457] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:54 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:55.086923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1226/2458] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:48:54 2022] POST /identity/v3/projects => generated 373 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:48:55.089678 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd7a090-cc07-4432-8900-d6ed4b75d974 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:55.104917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd7a090-cc07-4432-8900-d6ed4b75d974 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:55.191154 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b148f365-4484-4434-92b6-d692f84415d8 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:55.203500 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b148f365-4484-4434-92b6-d692f84415d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:55.206264 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314fb269-c921-466e-8090-57895424a18e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:55.207294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314fb269-c921-466e-8090-57895424a18e admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:55.207576 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314fb269-c921-466e-8090-57895424a18e admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:55.207852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314fb269-c921-466e-8090-57895424a18e admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:55.208485 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314fb269-c921-466e-8090-57895424a18e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:55.210506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314fb269-c921-466e-8090-57895424a18e admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:55.308943 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1233/2459] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:48:55 2022] POST /identity/v3/users => generated 287 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:48:55.314014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314fb269-c921-466e-8090-57895424a18e admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:55.326737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b148f365-4484-4434-92b6-d692f84415d8 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:55.334225 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314fb269-c921-466e-8090-57895424a18e admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:55.341041 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fbac10-0ca5-44db-9efd-ca6b901d7e4a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:55.341041 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fbac10-0ca5-44db-9efd-ca6b901d7e4a service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:55.341041 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fbac10-0ca5-44db-9efd-ca6b901d7e4a service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:55.341041 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fbac10-0ca5-44db-9efd-ca6b901d7e4a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:55.341407 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c2fbac10-0ca5-44db-9efd-ca6b901d7e4a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:55.349395 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe26407-479a-4e56-9b8a-1a3651c1df7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:55.350637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe26407-479a-4e56-9b8a-1a3651c1df7e admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:55.350824 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe26407-479a-4e56-9b8a-1a3651c1df7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:55.351238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe26407-479a-4e56-9b8a-1a3651c1df7e admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:55.351931 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe26407-479a-4e56-9b8a-1a3651c1df7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:55.357062 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe26407-479a-4e56-9b8a-1a3651c1df7e admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:55.371642 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1234/2460] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:55 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:48:55.386721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe26407-479a-4e56-9b8a-1a3651c1df7e admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:55.412082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe26407-479a-4e56-9b8a-1a3651c1df7e admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:55.421064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4519a1ef-3a8a-4c82-acbb-263239f92311 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:55.421842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4519a1ef-3a8a-4c82-acbb-263239f92311 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:55.422661 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4519a1ef-3a8a-4c82-acbb-263239f92311 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:55.422661 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4519a1ef-3a8a-4c82-acbb-263239f92311 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:55.423528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4519a1ef-3a8a-4c82-acbb-263239f92311 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:55.426909 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4519a1ef-3a8a-4c82-acbb-263239f92311 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:55.443929 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1235/2461] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:48:55 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:48:55.448364 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4519a1ef-3a8a-4c82-acbb-263239f92311 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:55.453237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2fbac10-0ca5-44db-9efd-ca6b901d7e4a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:55.456700 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2fbac10-0ca5-44db-9efd-ca6b901d7e4a service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:55.465391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4519a1ef-3a8a-4c82-acbb-263239f92311 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:55.468228 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1227/2462] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:48:55 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 278 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:48:55.472220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f1daf0-dc96-48f2-9969-086cfd447ad6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:55.473169 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f1daf0-dc96-48f2-9969-086cfd447ad6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:55.473622 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f1daf0-dc96-48f2-9969-086cfd447ad6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:55.479130 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f1daf0-dc96-48f2-9969-086cfd447ad6 admin admin] PATH_INFO: `/v3/projects/e1f45a76770149cd8d80b5e7cdd29b38/users/d48b81ec0c404f9c9a0137a2adda5759/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:55.495386 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f1daf0-dc96-48f2-9969-086cfd447ad6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1f45a76770149cd8d80b5e7cdd29b38, user_id=d48b81ec0c404f9c9a0137a2adda5759, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:55.499435 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f1daf0-dc96-48f2-9969-086cfd447ad6 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:55.659657 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1236/2463] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:48:55 2022] PUT /identity/v3/projects/e1f45a76770149cd8d80b5e7cdd29b38/users/d48b81ec0c404f9c9a0137a2adda5759/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:55.660171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d71be50-e452-421f-be6a-30a2c11e0976 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:55.660171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d71be50-e452-421f-be6a-30a2c11e0976 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:55.660171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d71be50-e452-421f-be6a-30a2c11e0976 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:55.710631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-8d71be50-e452-421f-be6a-30a2c11e0976 None None] MFA Rules not processed for user `d48b81ec0c404f9c9a0137a2adda5759`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:55.712080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8d71be50-e452-421f-be6a-30a2c11e0976 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:55.815124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1228/2464] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:55 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:48:55.830567 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f1daf0-dc96-48f2-9969-086cfd447ad6 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:55.832333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-99f1daf0-dc96-48f2-9969-086cfd447ad6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:55.946267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f1daf0-dc96-48f2-9969-086cfd447ad6 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:55.962050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c49a2e-a450-4777-828b-41c8d57195c7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:55.962852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c49a2e-a450-4777-828b-41c8d57195c7 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:55.963056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c49a2e-a450-4777-828b-41c8d57195c7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:55.963322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c49a2e-a450-4777-828b-41c8d57195c7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:55.972770 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c49a2e-a450-4777-828b-41c8d57195c7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:55.974495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c49a2e-a450-4777-828b-41c8d57195c7 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:55.984629 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1237/2465] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:48:55 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:48:56.015741 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d71be50-e452-421f-be6a-30a2c11e0976 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:56.085496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d71be50-e452-421f-be6a-30a2c11e0976 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:56.093849 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3fbd5e-4733-4b58-b8eb-2bf3dc2bf834 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:56.093849 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3fbd5e-4733-4b58-b8eb-2bf3dc2bf834 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:56.093849 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3fbd5e-4733-4b58-b8eb-2bf3dc2bf834 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:56.093849 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3fbd5e-4733-4b58-b8eb-2bf3dc2bf834 admin admin] PATH_INFO: `/v3/users/d48b81ec0c404f9c9a0137a2adda5759` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:56.095089 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3fbd5e-4733-4b58-b8eb-2bf3dc2bf834 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d48b81ec0c404f9c9a0137a2adda5759)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:56.097155 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3fbd5e-4733-4b58-b8eb-2bf3dc2bf834 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:56.142249 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=3,keystone:SELECT=95,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:56.299779 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ab3fbd5e-4733-4b58-b8eb-2bf3dc2bf834 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd48b81ec0c404f9c9a0137a2adda5759'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:56.300870 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ab3fbd5e-4733-4b58-b8eb-2bf3dc2bf834 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd48b81ec0c404f9c9a0137a2adda5759'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:56.302121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ab3fbd5e-4733-4b58-b8eb-2bf3dc2bf834 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd48b81ec0c404f9c9a0137a2adda5759'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:56.321753 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ab3fbd5e-4733-4b58-b8eb-2bf3dc2bf834 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd48b81ec0c404f9c9a0137a2adda5759'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:56.431310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ab3fbd5e-4733-4b58-b8eb-2bf3dc2bf834 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd48b81ec0c404f9c9a0137a2adda5759'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:56.481202 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0718629-7582-4637-991d-6cf3b209d8d5 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:56.481826 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0718629-7582-4637-991d-6cf3b209d8d5 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:56.482319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0718629-7582-4637-991d-6cf3b209d8d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:56.495330 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1229/2466] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:48:56 2022] DELETE /identity/v3/users/d48b81ec0c404f9c9a0137a2adda5759 => generated 0 bytes in 461 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:56.519336 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-d0718629-7582-4637-991d-6cf3b209d8d5 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:56.521189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d0718629-7582-4637-991d-6cf3b209d8d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:56.666250 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1238/2467] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:48:56.671497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:56.678429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3fbd5e-4733-4b58-b8eb-2bf3dc2bf834 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:56.678429 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ab3fbd5e-4733-4b58-b8eb-2bf3dc2bf834 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:56.769185 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3fbd5e-4733-4b58-b8eb-2bf3dc2bf834 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:56.774116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ef89e0-b347-4bf3-9ca0-edb9aa52cb29 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:56.775059 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ef89e0-b347-4bf3-9ca0-edb9aa52cb29 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:56.775539 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ef89e0-b347-4bf3-9ca0-edb9aa52cb29 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:56.776120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ef89e0-b347-4bf3-9ca0-edb9aa52cb29 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:56.791817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ef89e0-b347-4bf3-9ca0-edb9aa52cb29 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:56.794411 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ef89e0-b347-4bf3-9ca0-edb9aa52cb29 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:56.807990 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1230/2468] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:48:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:48:57.460681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0718629-7582-4637-991d-6cf3b209d8d5 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:48:57.462278 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d0718629-7582-4637-991d-6cf3b209d8d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:57.490030 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0718629-7582-4637-991d-6cf3b209d8d5 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:48:57.498060 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f2d08c-b8c9-48dc-9614-879efca031ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:48:57.498831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f2d08c-b8c9-48dc-9614-879efca031ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:57.499203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f2d08c-b8c9-48dc-9614-879efca031ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:57.499813 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f2d08c-b8c9-48dc-9614-879efca031ed admin admin] PATH_INFO: `/v3/projects/e1f45a76770149cd8d80b5e7cdd29b38` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:57.501490 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f2d08c-b8c9-48dc-9614-879efca031ed admin admin] RBAC: Authorizing `identity:delete_project(project_id=e1f45a76770149cd8d80b5e7cdd29b38)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:48:57.503870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f2d08c-b8c9-48dc-9614-879efca031ed admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:48:57.571430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:48:57.646614 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-51f2d08c-b8c9-48dc-9614-879efca031ed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e1f45a76770149cd8d80b5e7cdd29b38', 'request_id': 'req-51f2d08c-b8c9-48dc-9614-879efca031ed'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:57.672327 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-51f2d08c-b8c9-48dc-9614-879efca031ed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e1f45a76770149cd8d80b5e7cdd29b38', 'request_id': 'req-51f2d08c-b8c9-48dc-9614-879efca031ed'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:57.690100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-51f2d08c-b8c9-48dc-9614-879efca031ed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e1f45a76770149cd8d80b5e7cdd29b38', 'request_id': 'req-51f2d08c-b8c9-48dc-9614-879efca031ed'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:57.800808 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-51f2d08c-b8c9-48dc-9614-879efca031ed admin admin] The token cache is being invalidate because project e1f45a76770149cd8d80b5e7cdd29b38 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:48:57.801103 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-51f2d08c-b8c9-48dc-9614-879efca031ed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:57.801808 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-51f2d08c-b8c9-48dc-9614-879efca031ed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:48:57.807964 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1239/2469] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:48:57 2022] DELETE /identity/v3/projects/e1f45a76770149cd8d80b5e7cdd29b38 => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:48:59.772319 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73cfc8b-ffff-4464-a52f-ba22a7c75147 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:48:59.772817 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73cfc8b-ffff-4464-a52f-ba22a7c75147 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:48:59.772912 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73cfc8b-ffff-4464-a52f-ba22a7c75147 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:48:59.793765 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b73cfc8b-ffff-4464-a52f-ba22a7c75147 None None] MFA Rules not processed for user `20bee9f38eb14541a393037eaa389a44`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:48:59.795669 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b73cfc8b-ffff-4464-a52f-ba22a7c75147 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:48:59.863651 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1231/2470] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:48:59 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:49:07.797562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=53,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:49:09.855835 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=32 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:49:12.147273 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f2d08c-b8c9-48dc-9614-879efca031ed admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:12.148787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-51f2d08c-b8c9-48dc-9614-879efca031ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:12.222572 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f2d08c-b8c9-48dc-9614-879efca031ed admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:12.227783 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a38878a-6fce-4db3-bbbe-83a27ed1c954 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '49a3011c103b4e48904f876f81d27fba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:12.228410 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a38878a-6fce-4db3-bbbe-83a27ed1c954 service placement] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:12.228689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a38878a-6fce-4db3-bbbe-83a27ed1c954 service placement] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:12.228963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a38878a-6fce-4db3-bbbe-83a27ed1c954 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:12.230682 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0a38878a-6fce-4db3-bbbe-83a27ed1c954 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:12.242385 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a38878a-6fce-4db3-bbbe-83a27ed1c954 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:12.244071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a38878a-6fce-4db3-bbbe-83a27ed1c954 service placement] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:12.254250 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1240/2471] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:49:12 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:49:14.420655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8674cdac-9ae9-4b24-83e1-e3de7388c735 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:14.421121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8674cdac-9ae9-4b24-83e1-e3de7388c735 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:14.422202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8674cdac-9ae9-4b24-83e1-e3de7388c735 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:14.442385 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-8674cdac-9ae9-4b24-83e1-e3de7388c735 None None] MFA Rules not processed for user `7d6e0545a40a436d8c63820b795551a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:49:14.443622 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8674cdac-9ae9-4b24-83e1-e3de7388c735 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:14.497965 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1232/2472] 10.222.0.39 () {58 vars in 1009 bytes} [Fri Sep 30 09:49:14 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:49:14.518958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a38878a-6fce-4db3-bbbe-83a27ed1c954 service placement] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:14.520365 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0a38878a-6fce-4db3-bbbe-83a27ed1c954 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:14.588554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a38878a-6fce-4db3-bbbe-83a27ed1c954 service placement] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:14.593584 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7712b3c-207a-4824-9b38-54617f806be6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e5ad921e6fa41c297696ec7279f0fc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:14.594176 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7712b3c-207a-4824-9b38-54617f806be6 service swift] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:14.594453 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7712b3c-207a-4824-9b38-54617f806be6 service swift] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:14.594765 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7712b3c-207a-4824-9b38-54617f806be6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:14.603223 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7712b3c-207a-4824-9b38-54617f806be6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:14.605495 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7712b3c-207a-4824-9b38-54617f806be6 service swift] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:14.613747 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1241/2473] 10.222.0.39 () {60 vars in 1384 bytes} [Fri Sep 30 09:49:14 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 30 09:49:15.451953 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304c1618-7778-48a4-8f99-d419e8f35e32 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:15.452722 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304c1618-7778-48a4-8f99-d419e8f35e32 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:15.452722 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304c1618-7778-48a4-8f99-d419e8f35e32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:15.471526 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-304c1618-7778-48a4-8f99-d419e8f35e32 None None] MFA Rules not processed for user `7d6e0545a40a436d8c63820b795551a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:49:15.472723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-304c1618-7778-48a4-8f99-d419e8f35e32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:15.480208 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1233/2474] 10.222.0.39 () {58 vars in 967 bytes} [Fri Sep 30 09:49:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:49:15.493213 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7712b3c-207a-4824-9b38-54617f806be6 service swift] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:15.504669 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7712b3c-207a-4824-9b38-54617f806be6 service swift] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:15.510718 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0a0120-b309-4586-8cc6-d4f3dbb6ac2e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e5ad921e6fa41c297696ec7279f0fc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:15.511381 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0a0120-b309-4586-8cc6-d4f3dbb6ac2e service swift] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:15.524704 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0a0120-b309-4586-8cc6-d4f3dbb6ac2e service swift] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:15.524704 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0a0120-b309-4586-8cc6-d4f3dbb6ac2e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:15.524704 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0a0120-b309-4586-8cc6-d4f3dbb6ac2e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:15.525662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0a0120-b309-4586-8cc6-d4f3dbb6ac2e service swift] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:15.533249 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1242/2475] 10.222.0.39 () {60 vars in 1384 bytes} [Fri Sep 30 09:49:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 30 09:49:16.210653 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a207cdc8-405e-4290-8dac-54fbab347888 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:16.210942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a207cdc8-405e-4290-8dac-54fbab347888 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:16.211218 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a207cdc8-405e-4290-8dac-54fbab347888 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:16.231677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-a207cdc8-405e-4290-8dac-54fbab347888 None None] MFA Rules not processed for user `7d6e0545a40a436d8c63820b795551a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:49:16.232867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a207cdc8-405e-4290-8dac-54fbab347888 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:16.240013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1234/2476] 10.222.0.39 () {58 vars in 1009 bytes} [Fri Sep 30 09:49:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:49:16.252023 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0a0120-b309-4586-8cc6-d4f3dbb6ac2e service swift] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:16.262652 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0a0120-b309-4586-8cc6-d4f3dbb6ac2e service swift] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:16.268052 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4a2822-8370-4345-890c-1d7a34e397ae service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e5ad921e6fa41c297696ec7279f0fc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:16.268640 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4a2822-8370-4345-890c-1d7a34e397ae service swift] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:16.268924 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4a2822-8370-4345-890c-1d7a34e397ae service swift] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:16.269201 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4a2822-8370-4345-890c-1d7a34e397ae service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:16.277978 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4a2822-8370-4345-890c-1d7a34e397ae service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:16.279553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4a2822-8370-4345-890c-1d7a34e397ae service swift] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:16.287524 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1243/2477] 10.222.0.39 () {60 vars in 1384 bytes} [Fri Sep 30 09:49:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 30 09:49:16.843196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c03a8c-548a-47e7-a726-eab6982bbc4e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:16.843666 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c03a8c-548a-47e7-a726-eab6982bbc4e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:16.843761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c03a8c-548a-47e7-a726-eab6982bbc4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:16.862687 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c0c03a8c-548a-47e7-a726-eab6982bbc4e None None] MFA Rules not processed for user `7d6e0545a40a436d8c63820b795551a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:49:16.863930 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c0c03a8c-548a-47e7-a726-eab6982bbc4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:16.871819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1235/2478] 10.222.0.39 () {58 vars in 967 bytes} [Fri Sep 30 09:49:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:49:16.884172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4a2822-8370-4345-890c-1d7a34e397ae service swift] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:16.895353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4a2822-8370-4345-890c-1d7a34e397ae service swift] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:16.900295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94128052-1671-47af-be21-d5302e123444 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e5ad921e6fa41c297696ec7279f0fc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:16.900875 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94128052-1671-47af-be21-d5302e123444 service swift] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:16.901150 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94128052-1671-47af-be21-d5302e123444 service swift] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:16.901418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94128052-1671-47af-be21-d5302e123444 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:16.909499 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94128052-1671-47af-be21-d5302e123444 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:16.911100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94128052-1671-47af-be21-d5302e123444 service swift] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:16.918850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1244/2479] 10.222.0.39 () {60 vars in 1384 bytes} [Fri Sep 30 09:49:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 30 09:49:21.089679 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c03a8c-548a-47e7-a726-eab6982bbc4e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:21.091110 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c0c03a8c-548a-47e7-a726-eab6982bbc4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:21.145153 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c03a8c-548a-47e7-a726-eab6982bbc4e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:21.148743 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f929f12b-b006-4e27-a2ec-f3e511f91baa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:21.149826 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f929f12b-b006-4e27-a2ec-f3e511f91baa service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:21.149826 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f929f12b-b006-4e27-a2ec-f3e511f91baa service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:21.150027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f929f12b-b006-4e27-a2ec-f3e511f91baa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:21.152150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f929f12b-b006-4e27-a2ec-f3e511f91baa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:21.215607 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f929f12b-b006-4e27-a2ec-f3e511f91baa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:21.217076 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f929f12b-b006-4e27-a2ec-f3e511f91baa service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:21.228007 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1236/2480] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:49:21 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:49:21.385368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0563f5-9b27-48a5-83db-9d140e8403d3 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:21.386512 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0563f5-9b27-48a5-83db-9d140e8403d3 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:21.386628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0563f5-9b27-48a5-83db-9d140e8403d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:21.407015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-5a0563f5-9b27-48a5-83db-9d140e8403d3 None None] MFA Rules not processed for user `7d6e0545a40a436d8c63820b795551a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:49:21.408643 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-5a0563f5-9b27-48a5-83db-9d140e8403d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:21.416338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1245/2481] 10.222.0.39 () {58 vars in 1009 bytes} [Fri Sep 30 09:49:21 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:49:21.429603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f929f12b-b006-4e27-a2ec-f3e511f91baa service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:21.443147 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f929f12b-b006-4e27-a2ec-f3e511f91baa service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:21.449211 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da951052-a75d-4cc3-a0a7-02b71b338c06 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e5ad921e6fa41c297696ec7279f0fc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:21.449832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da951052-a75d-4cc3-a0a7-02b71b338c06 service swift] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:21.450114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da951052-a75d-4cc3-a0a7-02b71b338c06 service swift] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:21.450390 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da951052-a75d-4cc3-a0a7-02b71b338c06 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:21.463455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da951052-a75d-4cc3-a0a7-02b71b338c06 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:21.465124 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da951052-a75d-4cc3-a0a7-02b71b338c06 service swift] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:21.473799 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1237/2482] 10.222.0.39 () {60 vars in 1384 bytes} [Fri Sep 30 09:49:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 30 09:49:21.597904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0563f5-9b27-48a5-83db-9d140e8403d3 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:21.599378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-5a0563f5-9b27-48a5-83db-9d140e8403d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:21.638823 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:49:21.677056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0563f5-9b27-48a5-83db-9d140e8403d3 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:21.684090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bf7d60-2f09-4a95-836f-0405ea9db089 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:21.685107 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bf7d60-2f09-4a95-836f-0405ea9db089 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:21.685453 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bf7d60-2f09-4a95-836f-0405ea9db089 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:21.685736 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bf7d60-2f09-4a95-836f-0405ea9db089 admin admin] PATH_INFO: `/v3/users/6ebe000d3eb147aca44db7f213183c14` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:21.687202 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bf7d60-2f09-4a95-836f-0405ea9db089 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6ebe000d3eb147aca44db7f213183c14)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:21.689340 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bf7d60-2f09-4a95-836f-0405ea9db089 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:21.862129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-d2bf7d60-2f09-4a95-836f-0405ea9db089 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6ebe000d3eb147aca44db7f213183c14'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:49:21.882568 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-d2bf7d60-2f09-4a95-836f-0405ea9db089 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6ebe000d3eb147aca44db7f213183c14'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:49:21.883593 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-d2bf7d60-2f09-4a95-836f-0405ea9db089 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6ebe000d3eb147aca44db7f213183c14'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:49:21.953386 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-d2bf7d60-2f09-4a95-836f-0405ea9db089 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6ebe000d3eb147aca44db7f213183c14'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:49:21.965057 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-d2bf7d60-2f09-4a95-836f-0405ea9db089 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6ebe000d3eb147aca44db7f213183c14'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:49:21.973010 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1246/2483] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:49:21 2022] DELETE /identity/v3/users/6ebe000d3eb147aca44db7f213183c14 => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:49:21.977623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42e5971-cf47-4b3e-977b-f1f89af15b90 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:21.977906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42e5971-cf47-4b3e-977b-f1f89af15b90 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:21.978179 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42e5971-cf47-4b3e-977b-f1f89af15b90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:22.007738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c42e5971-cf47-4b3e-977b-f1f89af15b90 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:49:22.009247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c42e5971-cf47-4b3e-977b-f1f89af15b90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:22.043513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:49:22.106159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1238/2484] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:49:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:49:22.113994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bf7d60-2f09-4a95-836f-0405ea9db089 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:22.115631 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d2bf7d60-2f09-4a95-836f-0405ea9db089 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:22.252905 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bf7d60-2f09-4a95-836f-0405ea9db089 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:22.267063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66b9421-d269-40d2-a378-5e7d8d55aa0c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:22.267063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66b9421-d269-40d2-a378-5e7d8d55aa0c service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:22.267063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66b9421-d269-40d2-a378-5e7d8d55aa0c service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:22.267063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66b9421-d269-40d2-a378-5e7d8d55aa0c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:22.292074 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d66b9421-d269-40d2-a378-5e7d8d55aa0c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:22.294548 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d66b9421-d269-40d2-a378-5e7d8d55aa0c service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:22.320308 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1247/2485] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:49:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:49:22.872540 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42e5971-cf47-4b3e-977b-f1f89af15b90 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:22.874274 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c42e5971-cf47-4b3e-977b-f1f89af15b90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:22.898767 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42e5971-cf47-4b3e-977b-f1f89af15b90 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:22.908017 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0908dfcb-4990-401c-8ab8-4f15219e8418 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:22.908680 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0908dfcb-4990-401c-8ab8-4f15219e8418 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:22.908964 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0908dfcb-4990-401c-8ab8-4f15219e8418 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:22.909244 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0908dfcb-4990-401c-8ab8-4f15219e8418 admin admin] PATH_INFO: `/v3/projects/068a17bb811448d28e4ba1907f47b8a1` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:22.910426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0908dfcb-4990-401c-8ab8-4f15219e8418 admin admin] RBAC: Authorizing `identity:delete_project(project_id=068a17bb811448d28e4ba1907f47b8a1)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:22.912377 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0908dfcb-4990-401c-8ab8-4f15219e8418 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:23.067405 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0908dfcb-4990-401c-8ab8-4f15219e8418 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '068a17bb811448d28e4ba1907f47b8a1', 'request_id': 'req-0908dfcb-4990-401c-8ab8-4f15219e8418'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:49:23.074504 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0908dfcb-4990-401c-8ab8-4f15219e8418 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '068a17bb811448d28e4ba1907f47b8a1', 'request_id': 'req-0908dfcb-4990-401c-8ab8-4f15219e8418'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:49:23.081831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0908dfcb-4990-401c-8ab8-4f15219e8418 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '068a17bb811448d28e4ba1907f47b8a1', 'request_id': 'req-0908dfcb-4990-401c-8ab8-4f15219e8418'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:49:23.188349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0908dfcb-4990-401c-8ab8-4f15219e8418 admin admin] The token cache is being invalidate because project 068a17bb811448d28e4ba1907f47b8a1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:49:23.191962 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0908dfcb-4990-401c-8ab8-4f15219e8418 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:49:23.192324 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0908dfcb-4990-401c-8ab8-4f15219e8418 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:49:23.214635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1239/2486] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:49:22 2022] DELETE /identity/v3/projects/068a17bb811448d28e4ba1907f47b8a1 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:49:32.301068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=64,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:49:33.174297 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=44,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:49:39.826131 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a50a1ce-768d-41ff-a21e-5f8fc5a82486 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:39.826570 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a50a1ce-768d-41ff-a21e-5f8fc5a82486 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:39.826702 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a50a1ce-768d-41ff-a21e-5f8fc5a82486 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:39.846773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-1a50a1ce-768d-41ff-a21e-5f8fc5a82486 None None] MFA Rules not processed for user `7d6e0545a40a436d8c63820b795551a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:49:39.850743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-1a50a1ce-768d-41ff-a21e-5f8fc5a82486 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:39.910919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1248/2487] 10.222.0.39 () {58 vars in 967 bytes} [Fri Sep 30 09:49:39 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:49:39.924987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0908dfcb-4990-401c-8ab8-4f15219e8418 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:39.926536 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0908dfcb-4990-401c-8ab8-4f15219e8418 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:40.003316 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0908dfcb-4990-401c-8ab8-4f15219e8418 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:40.008425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7807c7d0-9db6-44d8-bacc-fcdd8e603b14 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e5ad921e6fa41c297696ec7279f0fc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:40.009208 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7807c7d0-9db6-44d8-bacc-fcdd8e603b14 service swift] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:40.009611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7807c7d0-9db6-44d8-bacc-fcdd8e603b14 service swift] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:40.009898 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7807c7d0-9db6-44d8-bacc-fcdd8e603b14 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:40.024717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7807c7d0-9db6-44d8-bacc-fcdd8e603b14 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:40.026363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7807c7d0-9db6-44d8-bacc-fcdd8e603b14 service swift] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:40.034923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1240/2488] 10.222.0.39 () {60 vars in 1384 bytes} [Fri Sep 30 09:49:39 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 30 09:49:43.066166 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb99d952-6a6b-4601-b37a-fb0cd8633a1c None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:43.067000 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb99d952-6a6b-4601-b37a-fb0cd8633a1c None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:43.067281 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb99d952-6a6b-4601-b37a-fb0cd8633a1c None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:43.068631 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1249/2489] 10.222.0.39 () {54 vars in 921 bytes} [Fri Sep 30 09:49:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:49:43.077351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664029a0-875e-4870-b6b1-cf534bc5ba32 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:43.077796 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664029a0-875e-4870-b6b1-cf534bc5ba32 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:43.078021 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664029a0-875e-4870-b6b1-cf534bc5ba32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:43.100024 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-664029a0-875e-4870-b6b1-cf534bc5ba32 None None] MFA Rules not processed for user `49a3011c103b4e48904f876f81d27fba`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:49:43.101243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-664029a0-875e-4870-b6b1-cf534bc5ba32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:43.165730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1241/2490] 10.222.0.39 () {58 vars in 1018 bytes} [Fri Sep 30 09:49:43 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:49:43.178552 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb99d952-6a6b-4601-b37a-fb0cd8633a1c None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:43.180331 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-cb99d952-6a6b-4601-b37a-fb0cd8633a1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:43.197602 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb99d952-6a6b-4601-b37a-fb0cd8633a1c None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:43.202899 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e19e8d-2b4f-43e0-9958-79bc39a1ce79 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '49a3011c103b4e48904f876f81d27fba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:43.203535 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e19e8d-2b4f-43e0-9958-79bc39a1ce79 service placement] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:43.203811 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e19e8d-2b4f-43e0-9958-79bc39a1ce79 service placement] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:43.204082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e19e8d-2b4f-43e0-9958-79bc39a1ce79 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:43.213138 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e19e8d-2b4f-43e0-9958-79bc39a1ce79 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:43.215607 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e19e8d-2b4f-43e0-9958-79bc39a1ce79 service placement] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:43.216753 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664029a0-875e-4870-b6b1-cf534bc5ba32 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:43.218690 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-664029a0-875e-4870-b6b1-cf534bc5ba32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:43.227748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1250/2491] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:49:43 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:49:43.287783 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664029a0-875e-4870-b6b1-cf534bc5ba32 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:43.292325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09533606-ea69-4261-928e-0cf5a9c58d43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:43.292949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09533606-ea69-4261-928e-0cf5a9c58d43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:43.293804 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09533606-ea69-4261-928e-0cf5a9c58d43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:43.294074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09533606-ea69-4261-928e-0cf5a9c58d43 admin admin] PATH_INFO: `/v3/users/587169e32cf64fe5b684e4b2ce0701a8` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:43.295215 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09533606-ea69-4261-928e-0cf5a9c58d43 admin admin] RBAC: Authorizing `identity:delete_user(user_id=587169e32cf64fe5b684e4b2ce0701a8)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:43.297122 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09533606-ea69-4261-928e-0cf5a9c58d43 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:43.397691 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:49:43.498318 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-09533606-ea69-4261-928e-0cf5a9c58d43 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '587169e32cf64fe5b684e4b2ce0701a8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:49:43.499865 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-09533606-ea69-4261-928e-0cf5a9c58d43 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '587169e32cf64fe5b684e4b2ce0701a8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:49:43.501137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-09533606-ea69-4261-928e-0cf5a9c58d43 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '587169e32cf64fe5b684e4b2ce0701a8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:49:43.510208 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-09533606-ea69-4261-928e-0cf5a9c58d43 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '587169e32cf64fe5b684e4b2ce0701a8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:49:43.584317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-09533606-ea69-4261-928e-0cf5a9c58d43 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '587169e32cf64fe5b684e4b2ce0701a8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:49:43.602630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1242/2492] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:49:43 2022] DELETE /identity/v3/users/587169e32cf64fe5b684e4b2ce0701a8 => generated 0 bytes in 387 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:49:43.711047 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e19e8d-2b4f-43e0-9958-79bc39a1ce79 service placement] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:43.712471 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-57e19e8d-2b4f-43e0-9958-79bc39a1ce79 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:43.770488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e19e8d-2b4f-43e0-9958-79bc39a1ce79 service placement] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:43.817072 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61da95a9-7ffd-400e-bd56-e5840dcf23b1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:43.817676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61da95a9-7ffd-400e-bd56-e5840dcf23b1 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:43.817953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61da95a9-7ffd-400e-bd56-e5840dcf23b1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:43.818222 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61da95a9-7ffd-400e-bd56-e5840dcf23b1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:43.819976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-61da95a9-7ffd-400e-bd56-e5840dcf23b1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:43.826343 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09533606-ea69-4261-928e-0cf5a9c58d43 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:43.828033 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-09533606-ea69-4261-928e-0cf5a9c58d43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:43.893555 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61da95a9-7ffd-400e-bd56-e5840dcf23b1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:43.895043 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61da95a9-7ffd-400e-bd56-e5840dcf23b1 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:43.898292 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09533606-ea69-4261-928e-0cf5a9c58d43 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:43.898929 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:49:43.903377 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b12336-b4be-405b-87e1-6eac7bd56665 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:43.904017 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b12336-b4be-405b-87e1-6eac7bd56665 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:43.904298 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b12336-b4be-405b-87e1-6eac7bd56665 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:43.904574 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b12336-b4be-405b-87e1-6eac7bd56665 admin admin] PATH_INFO: `/v3/projects/c1eb9ddbc0a84a308ceffeae5198f229` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:43.905095 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1251/2493] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:49:43 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:49:43.905965 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b12336-b4be-405b-87e1-6eac7bd56665 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c1eb9ddbc0a84a308ceffeae5198f229)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:43.907925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b12336-b4be-405b-87e1-6eac7bd56665 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:44.013555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-39b12336-b4be-405b-87e1-6eac7bd56665 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c1eb9ddbc0a84a308ceffeae5198f229', 'request_id': 'req-39b12336-b4be-405b-87e1-6eac7bd56665'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:49:44.019186 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-39b12336-b4be-405b-87e1-6eac7bd56665 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c1eb9ddbc0a84a308ceffeae5198f229', 'request_id': 'req-39b12336-b4be-405b-87e1-6eac7bd56665'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:49:44.025397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-39b12336-b4be-405b-87e1-6eac7bd56665 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c1eb9ddbc0a84a308ceffeae5198f229', 'request_id': 'req-39b12336-b4be-405b-87e1-6eac7bd56665'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:49:44.101210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-39b12336-b4be-405b-87e1-6eac7bd56665 admin admin] The token cache is being invalidate because project c1eb9ddbc0a84a308ceffeae5198f229 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:49:44.101860 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-39b12336-b4be-405b-87e1-6eac7bd56665 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:49:44.102244 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-39b12336-b4be-405b-87e1-6eac7bd56665 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:49:44.103933 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1243/2494] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:49:43 2022] DELETE /identity/v3/projects/c1eb9ddbc0a84a308ceffeae5198f229 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:49:44.109858 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9535ff-a402-40e4-8558-7548405c0960 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:44.116733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9535ff-a402-40e4-8558-7548405c0960 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:44.116733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9535ff-a402-40e4-8558-7548405c0960 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:44.130880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-2c9535ff-a402-40e4-8558-7548405c0960 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:49:44.132086 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-2c9535ff-a402-40e4-8558-7548405c0960 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:44.193637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1252/2495] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:49:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:49:44.202720 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74091980-095a-4733-937a-5d2280af3c7e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:44.203009 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74091980-095a-4733-937a-5d2280af3c7e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:44.203293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74091980-095a-4733-937a-5d2280af3c7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:44.223043 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-74091980-095a-4733-937a-5d2280af3c7e None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:49:44.224268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-74091980-095a-4733-937a-5d2280af3c7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:44.232846 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1244/2496] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:49:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:49:44.237128 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9535ff-a402-40e4-8558-7548405c0960 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:44.248312 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9535ff-a402-40e4-8558-7548405c0960 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:44.254117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8694c494-a0fc-4c0c-8d30-8174c934da7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:44.254749 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8694c494-a0fc-4c0c-8d30-8174c934da7c admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:44.255026 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8694c494-a0fc-4c0c-8d30-8174c934da7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:44.255293 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8694c494-a0fc-4c0c-8d30-8174c934da7c admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:44.255930 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8694c494-a0fc-4c0c-8d30-8174c934da7c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:49:44.256227 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8694c494-a0fc-4c0c-8d30-8174c934da7c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:44.257819 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8694c494-a0fc-4c0c-8d30-8174c934da7c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:44.269675 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1253/2497] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:49:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:49:44.274246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74091980-095a-4733-937a-5d2280af3c7e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:44.283896 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74091980-095a-4733-937a-5d2280af3c7e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:44.288625 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c0270e-378f-4fd1-80a9-0d7cc8374cfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:44.289245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c0270e-378f-4fd1-80a9-0d7cc8374cfa admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:44.289527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c0270e-378f-4fd1-80a9-0d7cc8374cfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:44.289796 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c0270e-378f-4fd1-80a9-0d7cc8374cfa admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:44.290425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c0270e-378f-4fd1-80a9-0d7cc8374cfa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:44.292362 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c0270e-378f-4fd1-80a9-0d7cc8374cfa admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:44.377895 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1245/2498] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:49:44 2022] POST /identity/v3/projects => generated 367 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:49:44.382019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8694c494-a0fc-4c0c-8d30-8174c934da7c admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:44.391842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8694c494-a0fc-4c0c-8d30-8174c934da7c admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:44.448692 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4acc6e-310b-4524-86df-7f1a974497b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:44.449371 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4acc6e-310b-4524-86df-7f1a974497b6 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:44.449650 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4acc6e-310b-4524-86df-7f1a974497b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:44.449920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4acc6e-310b-4524-86df-7f1a974497b6 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:44.450551 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4acc6e-310b-4524-86df-7f1a974497b6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:44.452492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4acc6e-310b-4524-86df-7f1a974497b6 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:44.608773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1254/2499] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:49:44 2022] POST /identity/v3/users => generated 284 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:49:44.609882 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c0270e-378f-4fd1-80a9-0d7cc8374cfa admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:44.620247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c0270e-378f-4fd1-80a9-0d7cc8374cfa admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:44.625179 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8a73b5-34c4-4118-bb19-81b6885556ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:44.625898 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8a73b5-34c4-4118-bb19-81b6885556ba admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:44.626173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8a73b5-34c4-4118-bb19-81b6885556ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:44.626446 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8a73b5-34c4-4118-bb19-81b6885556ba admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:44.627136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8a73b5-34c4-4118-bb19-81b6885556ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:44.628978 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8a73b5-34c4-4118-bb19-81b6885556ba admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:44.630318 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:49:44.640057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1246/2500] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:49:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:49:44.644073 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4acc6e-310b-4524-86df-7f1a974497b6 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:44.654614 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4acc6e-310b-4524-86df-7f1a974497b6 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:44.659232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df98695-cdd8-45e8-9d5d-7c997776307e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:44.659826 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df98695-cdd8-45e8-9d5d-7c997776307e admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:44.660164 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df98695-cdd8-45e8-9d5d-7c997776307e admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:44.660380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df98695-cdd8-45e8-9d5d-7c997776307e admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:44.661028 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df98695-cdd8-45e8-9d5d-7c997776307e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:44.662648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df98695-cdd8-45e8-9d5d-7c997776307e admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:44.674072 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1255/2501] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:49:44 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:49:44.678085 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8a73b5-34c4-4118-bb19-81b6885556ba admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:44.687951 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8a73b5-34c4-4118-bb19-81b6885556ba admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:44.692542 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed2b072-1156-4ced-bfe3-7f522b6fd5c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:44.694037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed2b072-1156-4ced-bfe3-7f522b6fd5c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:44.694037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed2b072-1156-4ced-bfe3-7f522b6fd5c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:44.694037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed2b072-1156-4ced-bfe3-7f522b6fd5c7 admin admin] PATH_INFO: `/v3/projects/f38d95783f784d68806d1b2f1d5c4b1c/users/62835e5630314e42bf5ddf443dec8055/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:44.709140 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed2b072-1156-4ced-bfe3-7f522b6fd5c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f38d95783f784d68806d1b2f1d5c4b1c, user_id=62835e5630314e42bf5ddf443dec8055, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:44.711977 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed2b072-1156-4ced-bfe3-7f522b6fd5c7 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:44.794025 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1247/2502] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:49:44 2022] PUT /identity/v3/projects/f38d95783f784d68806d1b2f1d5c4b1c/users/62835e5630314e42bf5ddf443dec8055/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:49:44.804442 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d7b6e0-9687-4793-847a-68d2af794799 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:44.808014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d7b6e0-9687-4793-847a-68d2af794799 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:44.808014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d7b6e0-9687-4793-847a-68d2af794799 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:44.836666 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-83d7b6e0-9687-4793-847a-68d2af794799 None None] MFA Rules not processed for user `62835e5630314e42bf5ddf443dec8055`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:49:44.836666 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-83d7b6e0-9687-4793-847a-68d2af794799 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:44.890752 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:49:44.900356 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1256/2503] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:49:44 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:49:44.912884 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed2b072-1156-4ced-bfe3-7f522b6fd5c7 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:49:44.916901 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9ed2b072-1156-4ced-bfe3-7f522b6fd5c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:49:44.972247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed2b072-1156-4ced-bfe3-7f522b6fd5c7 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:49:44.976025 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94926d45-71d8-4bda-a683-ce542043467d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf907d4033064fb8a2707b1bbb0a180b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:49:44.976637 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94926d45-71d8-4bda-a683-ce542043467d service glance] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:49:44.976913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94926d45-71d8-4bda-a683-ce542043467d service glance] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:49:44.977179 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94926d45-71d8-4bda-a683-ce542043467d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:49:44.985758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94926d45-71d8-4bda-a683-ce542043467d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:49:44.987333 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94926d45-71d8-4bda-a683-ce542043467d service glance] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:49:44.996666 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1248/2504] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:49:44 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:49:54.992408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=38,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:49:55.010165 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:50:20.014930 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9130d39a-2bb6-4e33-a7be-0a43dccbc411 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:50:20.015586 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9130d39a-2bb6-4e33-a7be-0a43dccbc411 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:50:20.015586 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9130d39a-2bb6-4e33-a7be-0a43dccbc411 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:50:20.034684 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-9130d39a-2bb6-4e33-a7be-0a43dccbc411 None None] MFA Rules not processed for user `7ad21da05fc246d489b0afa641f20216`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:50:20.035870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-9130d39a-2bb6-4e33-a7be-0a43dccbc411 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:50:20.091224 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1257/2505] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:50:20 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:50:23.489518 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94926d45-71d8-4bda-a683-ce542043467d service glance] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:50:23.561367 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94926d45-71d8-4bda-a683-ce542043467d service glance] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:50:23.566700 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69454647-f33e-48f2-ad0b-5b8fd1712ca7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:50:23.567397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69454647-f33e-48f2-ad0b-5b8fd1712ca7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:50:23.567709 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69454647-f33e-48f2-ad0b-5b8fd1712ca7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:50:23.567985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69454647-f33e-48f2-ad0b-5b8fd1712ca7 admin admin] PATH_INFO: `/v3/users/62835e5630314e42bf5ddf443dec8055` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:50:23.569387 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69454647-f33e-48f2-ad0b-5b8fd1712ca7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=62835e5630314e42bf5ddf443dec8055)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:50:23.571515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69454647-f33e-48f2-ad0b-5b8fd1712ca7 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:50:23.755012 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-69454647-f33e-48f2-ad0b-5b8fd1712ca7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '62835e5630314e42bf5ddf443dec8055'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:50:23.755725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-69454647-f33e-48f2-ad0b-5b8fd1712ca7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '62835e5630314e42bf5ddf443dec8055'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:50:23.756372 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-69454647-f33e-48f2-ad0b-5b8fd1712ca7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '62835e5630314e42bf5ddf443dec8055'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:50:23.765029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-69454647-f33e-48f2-ad0b-5b8fd1712ca7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '62835e5630314e42bf5ddf443dec8055'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:50:23.861942 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-69454647-f33e-48f2-ad0b-5b8fd1712ca7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '62835e5630314e42bf5ddf443dec8055'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:50:23.882418 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1249/2506] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:50:23 2022] DELETE /identity/v3/users/62835e5630314e42bf5ddf443dec8055 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:50:23.886985 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de0308a-397b-4c2e-9294-ce9aae76feb4 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:50:23.887514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de0308a-397b-4c2e-9294-ce9aae76feb4 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:50:23.887514 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de0308a-397b-4c2e-9294-ce9aae76feb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:50:23.907133 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-5de0308a-397b-4c2e-9294-ce9aae76feb4 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:50:23.908978 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-5de0308a-397b-4c2e-9294-ce9aae76feb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:50:23.967485 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1258/2507] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:50:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:50:23.977850 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69454647-f33e-48f2-ad0b-5b8fd1712ca7 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:50:23.983168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-69454647-f33e-48f2-ad0b-5b8fd1712ca7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:50:24.038122 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69454647-f33e-48f2-ad0b-5b8fd1712ca7 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:50:24.042013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b25f776-ac44-417a-8f08-8cca7bc10ee2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:50:24.042646 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b25f776-ac44-417a-8f08-8cca7bc10ee2 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:50:24.042926 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b25f776-ac44-417a-8f08-8cca7bc10ee2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:50:24.043197 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b25f776-ac44-417a-8f08-8cca7bc10ee2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:50:24.053641 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b25f776-ac44-417a-8f08-8cca7bc10ee2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:50:24.055449 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b25f776-ac44-417a-8f08-8cca7bc10ee2 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:50:24.069692 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1250/2508] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:50:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:50:24.135782 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dfb58d-644e-46e2-93e2-54b996edde94 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:50:24.136075 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dfb58d-644e-46e2-93e2-54b996edde94 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:50:24.136352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dfb58d-644e-46e2-93e2-54b996edde94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:50:24.159611 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-91dfb58d-644e-46e2-93e2-54b996edde94 None None] MFA Rules not processed for user `8c4ced79c3ff4842b30b472acb555cd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:50:24.160818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-91dfb58d-644e-46e2-93e2-54b996edde94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:50:24.221220 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1259/2509] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:50:24 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:50:24.508440 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b25f776-ac44-417a-8f08-8cca7bc10ee2 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:50:24.510013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5b25f776-ac44-417a-8f08-8cca7bc10ee2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:50:24.529738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b25f776-ac44-417a-8f08-8cca7bc10ee2 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:50:24.534680 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde0f6b8-8bfe-458b-abaf-56eb6945d62c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:50:24.535377 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde0f6b8-8bfe-458b-abaf-56eb6945d62c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:50:24.535611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde0f6b8-8bfe-458b-abaf-56eb6945d62c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:50:24.535896 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde0f6b8-8bfe-458b-abaf-56eb6945d62c admin admin] PATH_INFO: `/v3/projects/f38d95783f784d68806d1b2f1d5c4b1c` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:50:24.537049 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde0f6b8-8bfe-458b-abaf-56eb6945d62c admin admin] RBAC: Authorizing `identity:delete_project(project_id=f38d95783f784d68806d1b2f1d5c4b1c)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:50:24.539050 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde0f6b8-8bfe-458b-abaf-56eb6945d62c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:50:24.558082 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:50:24.671520 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cde0f6b8-8bfe-458b-abaf-56eb6945d62c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f38d95783f784d68806d1b2f1d5c4b1c', 'request_id': 'req-cde0f6b8-8bfe-458b-abaf-56eb6945d62c'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:50:24.680063 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cde0f6b8-8bfe-458b-abaf-56eb6945d62c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f38d95783f784d68806d1b2f1d5c4b1c', 'request_id': 'req-cde0f6b8-8bfe-458b-abaf-56eb6945d62c'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:50:24.691694 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cde0f6b8-8bfe-458b-abaf-56eb6945d62c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f38d95783f784d68806d1b2f1d5c4b1c', 'request_id': 'req-cde0f6b8-8bfe-458b-abaf-56eb6945d62c'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:50:24.765237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cde0f6b8-8bfe-458b-abaf-56eb6945d62c admin admin] The token cache is being invalidate because project f38d95783f784d68806d1b2f1d5c4b1c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:50:24.765530 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cde0f6b8-8bfe-458b-abaf-56eb6945d62c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:50:24.766164 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cde0f6b8-8bfe-458b-abaf-56eb6945d62c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:50:24.771505 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1251/2510] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:50:24 2022] DELETE /identity/v3/projects/f38d95783f784d68806d1b2f1d5c4b1c => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:50:24.773300 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fb1b53-22bb-4534-8c4e-fba61f14bc21 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:50:24.773596 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fb1b53-22bb-4534-8c4e-fba61f14bc21 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:50:24.773891 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fb1b53-22bb-4534-8c4e-fba61f14bc21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:50:24.794475 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-90fb1b53-22bb-4534-8c4e-fba61f14bc21 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:50:24.796010 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-90fb1b53-22bb-4534-8c4e-fba61f14bc21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:50:24.846641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:50:24.862147 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1260/2511] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:50:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:50:24.930879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798f4b2e-fe1d-46d3-a423-3820e9fe0bbd None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:50:24.931272 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798f4b2e-fe1d-46d3-a423-3820e9fe0bbd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:50:24.931618 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798f4b2e-fe1d-46d3-a423-3820e9fe0bbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:50:24.953144 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-798f4b2e-fe1d-46d3-a423-3820e9fe0bbd None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:50:24.954561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-798f4b2e-fe1d-46d3-a423-3820e9fe0bbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:50:24.962816 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1252/2512] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:50:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:50:24.967215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fb1b53-22bb-4534-8c4e-fba61f14bc21 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:50:24.986045 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fb1b53-22bb-4534-8c4e-fba61f14bc21 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:50:24.992967 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cc1bf2-8b00-4a2a-9457-6e2f3820ba13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:50:24.994388 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cc1bf2-8b00-4a2a-9457-6e2f3820ba13 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:50:24.995173 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cc1bf2-8b00-4a2a-9457-6e2f3820ba13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:50:24.995173 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cc1bf2-8b00-4a2a-9457-6e2f3820ba13 admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:50:24.996429 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09cc1bf2-8b00-4a2a-9457-6e2f3820ba13 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:50:24.996837 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09cc1bf2-8b00-4a2a-9457-6e2f3820ba13 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:50:24.998719 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09cc1bf2-8b00-4a2a-9457-6e2f3820ba13 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:50:25.014637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1261/2513] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:50:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:50:25.016794 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798f4b2e-fe1d-46d3-a423-3820e9fe0bbd None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:50:25.026844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798f4b2e-fe1d-46d3-a423-3820e9fe0bbd None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:50:25.031859 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af71da63-09ff-4565-a5e8-a089b9e69839 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:50:25.032554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af71da63-09ff-4565-a5e8-a089b9e69839 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:50:25.033419 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af71da63-09ff-4565-a5e8-a089b9e69839 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:50:25.033761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af71da63-09ff-4565-a5e8-a089b9e69839 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:50:25.034446 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af71da63-09ff-4565-a5e8-a089b9e69839 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:50:25.036435 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af71da63-09ff-4565-a5e8-a089b9e69839 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:50:25.145484 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1253/2514] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:50:25 2022] POST /identity/v3/projects => generated 383 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:50:25.149610 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cc1bf2-8b00-4a2a-9457-6e2f3820ba13 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:50:25.160118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cc1bf2-8b00-4a2a-9457-6e2f3820ba13 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:50:25.222264 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f454beef-4ff1-4f5b-a727-18fad739c67b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:50:25.222922 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f454beef-4ff1-4f5b-a727-18fad739c67b admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:50:25.223201 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f454beef-4ff1-4f5b-a727-18fad739c67b admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:50:25.223476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f454beef-4ff1-4f5b-a727-18fad739c67b admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:50:25.224118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f454beef-4ff1-4f5b-a727-18fad739c67b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:50:25.226055 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f454beef-4ff1-4f5b-a727-18fad739c67b admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:50:25.334786 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1262/2515] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:50:25 2022] POST /identity/v3/users => generated 292 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:50:25.339174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af71da63-09ff-4565-a5e8-a089b9e69839 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:50:25.350230 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af71da63-09ff-4565-a5e8-a089b9e69839 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:50:25.355024 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd13b65-75fc-4c08-a1de-4672fb042bdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:50:25.357090 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd13b65-75fc-4c08-a1de-4672fb042bdf admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:50:25.357280 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd13b65-75fc-4c08-a1de-4672fb042bdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:50:25.357561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd13b65-75fc-4c08-a1de-4672fb042bdf admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:50:25.358264 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd13b65-75fc-4c08-a1de-4672fb042bdf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:50:25.359970 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd13b65-75fc-4c08-a1de-4672fb042bdf admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:50:25.371490 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1254/2516] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:50:25 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:50:25.375442 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f454beef-4ff1-4f5b-a727-18fad739c67b admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:50:25.386036 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f454beef-4ff1-4f5b-a727-18fad739c67b admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:50:25.391156 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0d411c-5101-4beb-82e7-7e2cefa09b1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:50:25.391822 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0d411c-5101-4beb-82e7-7e2cefa09b1d admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:50:25.392197 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0d411c-5101-4beb-82e7-7e2cefa09b1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:50:25.393049 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0d411c-5101-4beb-82e7-7e2cefa09b1d admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:50:25.393760 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a0d411c-5101-4beb-82e7-7e2cefa09b1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:50:25.395572 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a0d411c-5101-4beb-82e7-7e2cefa09b1d admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:50:25.407773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1263/2517] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:50:25 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:50:25.412414 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd13b65-75fc-4c08-a1de-4672fb042bdf admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:50:25.424574 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd13b65-75fc-4c08-a1de-4672fb042bdf admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:50:25.431313 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39647c1-4b18-4426-9f93-44998e7254b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:50:25.432083 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39647c1-4b18-4426-9f93-44998e7254b3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:50:25.432485 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39647c1-4b18-4426-9f93-44998e7254b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:50:25.432828 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39647c1-4b18-4426-9f93-44998e7254b3 admin admin] PATH_INFO: `/v3/projects/b3db64185ae94cefa4efa6f3e03c2f7e/users/8f8db95b40ca4350b4f86ce9f1593194/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:50:25.450902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39647c1-4b18-4426-9f93-44998e7254b3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3db64185ae94cefa4efa6f3e03c2f7e, user_id=8f8db95b40ca4350b4f86ce9f1593194, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:50:25.453845 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39647c1-4b18-4426-9f93-44998e7254b3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:50:25.571293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1255/2518] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:50:25 2022] PUT /identity/v3/projects/b3db64185ae94cefa4efa6f3e03c2f7e/users/8f8db95b40ca4350b4f86ce9f1593194/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:50:25.576799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef0d579-12cd-4a45-b1d7-5836e8479047 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:50:25.577611 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef0d579-12cd-4a45-b1d7-5836e8479047 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:50:25.578107 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef0d579-12cd-4a45-b1d7-5836e8479047 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:50:25.605452 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-6ef0d579-12cd-4a45-b1d7-5836e8479047 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:50:25.606818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-6ef0d579-12cd-4a45-b1d7-5836e8479047 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:50:25.673154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1264/2519] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:50:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:50:25.683762 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39647c1-4b18-4426-9f93-44998e7254b3 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:50:25.684943 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c39647c1-4b18-4426-9f93-44998e7254b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:50:25.748902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39647c1-4b18-4426-9f93-44998e7254b3 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:50:25.755423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8b0bd8-a79e-437e-aaaf-5231ea867560 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:50:25.756240 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8b0bd8-a79e-437e-aaaf-5231ea867560 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:50:25.756631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8b0bd8-a79e-437e-aaaf-5231ea867560 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:50:25.757016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8b0bd8-a79e-437e-aaaf-5231ea867560 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:50:25.768435 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8b0bd8-a79e-437e-aaaf-5231ea867560 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:50:25.770187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8b0bd8-a79e-437e-aaaf-5231ea867560 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:50:25.780948 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1256/2520] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:50:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:50:27.187739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f890b85f-b0de-4abf-8c1e-2daa9c9a2080 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:50:27.188285 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f890b85f-b0de-4abf-8c1e-2daa9c9a2080 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:50:27.188381 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f890b85f-b0de-4abf-8c1e-2daa9c9a2080 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:50:27.217850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-f890b85f-b0de-4abf-8c1e-2daa9c9a2080 None None] MFA Rules not processed for user `8f8db95b40ca4350b4f86ce9f1593194`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:50:27.219136 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-f890b85f-b0de-4abf-8c1e-2daa9c9a2080 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:50:27.277875 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:50:27.297289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1265/2521] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:50:27 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:50:27.308300 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8b0bd8-a79e-437e-aaaf-5231ea867560 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:50:27.309715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ec8b0bd8-a79e-437e-aaaf-5231ea867560 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:50:27.356715 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:50:27.367403 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8b0bd8-a79e-437e-aaaf-5231ea867560 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:50:27.371326 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c839a3f-092b-416f-936b-39dcbbe1549d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:50:27.381239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c839a3f-092b-416f-936b-39dcbbe1549d service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:50:27.381239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c839a3f-092b-416f-936b-39dcbbe1549d service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:50:27.381239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c839a3f-092b-416f-936b-39dcbbe1549d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:50:27.382317 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c839a3f-092b-416f-936b-39dcbbe1549d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:50:27.383781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c839a3f-092b-416f-936b-39dcbbe1549d service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:50:27.393452 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1257/2522] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:50:27 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:50:29.225516 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ad632a-16e9-4fea-8868-1c5d3dac7290 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:50:29.226025 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ad632a-16e9-4fea-8868-1c5d3dac7290 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:50:29.226122 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ad632a-16e9-4fea-8868-1c5d3dac7290 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:50:29.247215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-33ad632a-16e9-4fea-8868-1c5d3dac7290 None None] MFA Rules not processed for user `8f8db95b40ca4350b4f86ce9f1593194`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:50:29.248445 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-33ad632a-16e9-4fea-8868-1c5d3dac7290 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:50:29.256260 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1266/2523] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:50:29 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:50:37.812475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=6 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:50:39.241726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=9 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:50:57.503143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c839a3f-092b-416f-936b-39dcbbe1549d service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:50:57.512353 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c839a3f-092b-416f-936b-39dcbbe1549d service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:50:57.515997 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e6ece0-e1ba-4429-a79a-e5b4d1c084e5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:50:57.516606 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e6ece0-e1ba-4429-a79a-e5b4d1c084e5 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:50:57.516881 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e6ece0-e1ba-4429-a79a-e5b4d1c084e5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:50:57.517148 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e6ece0-e1ba-4429-a79a-e5b4d1c084e5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:50:57.519090 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-83e6ece0-e1ba-4429-a79a-e5b4d1c084e5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:50:57.595663 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e6ece0-e1ba-4429-a79a-e5b4d1c084e5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:50:57.597131 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e6ece0-e1ba-4429-a79a-e5b4d1c084e5 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:50:57.608943 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1258/2524] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:50:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:51:00.117355 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ad632a-16e9-4fea-8868-1c5d3dac7290 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:00.127199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ad632a-16e9-4fea-8868-1c5d3dac7290 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:00.130774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec51c49-1dae-4ba3-99e7-6b57ad242e60 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:00.131358 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec51c49-1dae-4ba3-99e7-6b57ad242e60 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:00.131636 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec51c49-1dae-4ba3-99e7-6b57ad242e60 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:00.131906 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec51c49-1dae-4ba3-99e7-6b57ad242e60 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:00.133598 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-aec51c49-1dae-4ba3-99e7-6b57ad242e60 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:00.146400 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec51c49-1dae-4ba3-99e7-6b57ad242e60 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:00.147858 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec51c49-1dae-4ba3-99e7-6b57ad242e60 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:00.158428 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1267/2525] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:51:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:51:06.624982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed154d5-79bf-422f-bcc5-d7f856719db1 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:06.624982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed154d5-79bf-422f-bcc5-d7f856719db1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:06.624982 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed154d5-79bf-422f-bcc5-d7f856719db1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:06.643843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-8ed154d5-79bf-422f-bcc5-d7f856719db1 None None] MFA Rules not processed for user `7d6e0545a40a436d8c63820b795551a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:51:06.645236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8ed154d5-79bf-422f-bcc5-d7f856719db1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:06.703341 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1259/2526] 10.222.0.39 () {58 vars in 967 bytes} [Fri Sep 30 09:51:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:51:06.866700 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec51c49-1dae-4ba3-99e7-6b57ad242e60 service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:06.868108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-aec51c49-1dae-4ba3-99e7-6b57ad242e60 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:06.935375 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec51c49-1dae-4ba3-99e7-6b57ad242e60 service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:06.940318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9714e7fb-dc69-4f71-b971-7878833557af service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e5ad921e6fa41c297696ec7279f0fc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:06.940915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9714e7fb-dc69-4f71-b971-7878833557af service swift] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:06.941190 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9714e7fb-dc69-4f71-b971-7878833557af service swift] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:06.941460 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9714e7fb-dc69-4f71-b971-7878833557af service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:06.949666 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9714e7fb-dc69-4f71-b971-7878833557af service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:06.951280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9714e7fb-dc69-4f71-b971-7878833557af service swift] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:06.959156 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1268/2527] 10.222.0.39 () {60 vars in 1384 bytes} [Fri Sep 30 09:51:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 30 09:51:13.083208 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed154d5-79bf-422f-bcc5-d7f856719db1 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:13.093746 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed154d5-79bf-422f-bcc5-d7f856719db1 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:13.097679 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c099d892-eeb8-4fea-8c0d-f32ba3447633 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:13.098293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c099d892-eeb8-4fea-8c0d-f32ba3447633 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:13.098574 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c099d892-eeb8-4fea-8c0d-f32ba3447633 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:13.098878 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c099d892-eeb8-4fea-8c0d-f32ba3447633 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:13.100642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c099d892-eeb8-4fea-8c0d-f32ba3447633 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:13.178149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c099d892-eeb8-4fea-8c0d-f32ba3447633 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:13.180310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c099d892-eeb8-4fea-8c0d-f32ba3447633 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:13.191120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1260/2528] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:51:13 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:51:15.010842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9714e7fb-dc69-4f71-b971-7878833557af service swift] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:15.012307 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-9714e7fb-dc69-4f71-b971-7878833557af service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:15.026636 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9714e7fb-dc69-4f71-b971-7878833557af service swift] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:15.036916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65455f1-63e4-4908-a430-d589a7936ac7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:15.040785 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65455f1-63e4-4908-a430-d589a7936ac7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:15.040785 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65455f1-63e4-4908-a430-d589a7936ac7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:15.040785 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65455f1-63e4-4908-a430-d589a7936ac7 admin admin] PATH_INFO: `/v3/users/4d46ec34caf74ea7b7c8f836c2f7ea6c` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:15.040785 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65455f1-63e4-4908-a430-d589a7936ac7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4d46ec34caf74ea7b7c8f836c2f7ea6c)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:15.041563 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65455f1-63e4-4908-a430-d589a7936ac7 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:15.262877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-b65455f1-63e4-4908-a430-d589a7936ac7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4d46ec34caf74ea7b7c8f836c2f7ea6c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:15.275607 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-b65455f1-63e4-4908-a430-d589a7936ac7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4d46ec34caf74ea7b7c8f836c2f7ea6c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:15.276277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-b65455f1-63e4-4908-a430-d589a7936ac7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4d46ec34caf74ea7b7c8f836c2f7ea6c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:15.360688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-b65455f1-63e4-4908-a430-d589a7936ac7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4d46ec34caf74ea7b7c8f836c2f7ea6c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:15.368725 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-b65455f1-63e4-4908-a430-d589a7936ac7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4d46ec34caf74ea7b7c8f836c2f7ea6c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:15.375086 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1269/2529] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:51:15 2022] DELETE /identity/v3/users/4d46ec34caf74ea7b7c8f836c2f7ea6c => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:51:15.652584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c099d892-eeb8-4fea-8c0d-f32ba3447633 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:15.664122 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c099d892-eeb8-4fea-8c0d-f32ba3447633 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:15.689675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:51:15.720406 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76613601-7920-4255-a579-9075afd00ede admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:15.721042 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76613601-7920-4255-a579-9075afd00ede admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:15.721373 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76613601-7920-4255-a579-9075afd00ede admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:15.721600 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76613601-7920-4255-a579-9075afd00ede admin admin] PATH_INFO: `/v3/projects/0be64f8f1d4b452e84a7bcd64f6100d1` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:15.722758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76613601-7920-4255-a579-9075afd00ede admin admin] RBAC: Authorizing `identity:delete_project(project_id=0be64f8f1d4b452e84a7bcd64f6100d1)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:15.724707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76613601-7920-4255-a579-9075afd00ede admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:15.834009 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-76613601-7920-4255-a579-9075afd00ede admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0be64f8f1d4b452e84a7bcd64f6100d1', 'request_id': 'req-76613601-7920-4255-a579-9075afd00ede'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:15.841109 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-76613601-7920-4255-a579-9075afd00ede admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0be64f8f1d4b452e84a7bcd64f6100d1', 'request_id': 'req-76613601-7920-4255-a579-9075afd00ede'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:15.846925 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-76613601-7920-4255-a579-9075afd00ede admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0be64f8f1d4b452e84a7bcd64f6100d1', 'request_id': 'req-76613601-7920-4255-a579-9075afd00ede'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:15.917361 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-76613601-7920-4255-a579-9075afd00ede admin admin] The token cache is being invalidate because project 0be64f8f1d4b452e84a7bcd64f6100d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:51:15.917655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-76613601-7920-4255-a579-9075afd00ede admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:15.918285 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-76613601-7920-4255-a579-9075afd00ede admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:15.919931 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1261/2530] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:51:15 2022] DELETE /identity/v3/projects/0be64f8f1d4b452e84a7bcd64f6100d1 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:51:15.924025 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65455f1-63e4-4908-a430-d589a7936ac7 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:15.925465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b65455f1-63e4-4908-a430-d589a7936ac7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:15.938689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65455f1-63e4-4908-a430-d589a7936ac7 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:15.984046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:51:15.995216 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa894b5f-bc92-4899-9f5a-3f11b7451806 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:15.995838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa894b5f-bc92-4899-9f5a-3f11b7451806 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:15.996205 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa894b5f-bc92-4899-9f5a-3f11b7451806 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:15.996365 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa894b5f-bc92-4899-9f5a-3f11b7451806 admin admin] PATH_INFO: `/v3/users/7166a00d45684b51aa90afa1b05ca7d8` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:15.997468 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa894b5f-bc92-4899-9f5a-3f11b7451806 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7166a00d45684b51aa90afa1b05ca7d8)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:15.999687 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa894b5f-bc92-4899-9f5a-3f11b7451806 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:16.165569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-fa894b5f-bc92-4899-9f5a-3f11b7451806 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7166a00d45684b51aa90afa1b05ca7d8'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:16.177690 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-fa894b5f-bc92-4899-9f5a-3f11b7451806 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7166a00d45684b51aa90afa1b05ca7d8'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:16.178420 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-fa894b5f-bc92-4899-9f5a-3f11b7451806 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7166a00d45684b51aa90afa1b05ca7d8'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:16.249369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-fa894b5f-bc92-4899-9f5a-3f11b7451806 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7166a00d45684b51aa90afa1b05ca7d8'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:16.256590 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-fa894b5f-bc92-4899-9f5a-3f11b7451806 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7166a00d45684b51aa90afa1b05ca7d8'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:16.263266 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1270/2531] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:51:15 2022] DELETE /identity/v3/users/7166a00d45684b51aa90afa1b05ca7d8 => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:51:16.519037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76613601-7920-4255-a579-9075afd00ede admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:16.522731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-76613601-7920-4255-a579-9075afd00ede admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:16.533840 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76613601-7920-4255-a579-9075afd00ede admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:16.591677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897a42c7-69a0-4303-85ed-de7b9cd78a07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:16.592312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897a42c7-69a0-4303-85ed-de7b9cd78a07 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:16.592595 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897a42c7-69a0-4303-85ed-de7b9cd78a07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:16.592871 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897a42c7-69a0-4303-85ed-de7b9cd78a07 admin admin] PATH_INFO: `/v3/projects/ae05d51a1efc4c5398e9fa4ffeb7faa0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:16.593984 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897a42c7-69a0-4303-85ed-de7b9cd78a07 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ae05d51a1efc4c5398e9fa4ffeb7faa0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:16.595966 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897a42c7-69a0-4303-85ed-de7b9cd78a07 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:16.711979 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-897a42c7-69a0-4303-85ed-de7b9cd78a07 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ae05d51a1efc4c5398e9fa4ffeb7faa0', 'request_id': 'req-897a42c7-69a0-4303-85ed-de7b9cd78a07'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:16.717422 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-897a42c7-69a0-4303-85ed-de7b9cd78a07 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ae05d51a1efc4c5398e9fa4ffeb7faa0', 'request_id': 'req-897a42c7-69a0-4303-85ed-de7b9cd78a07'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:16.719282 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:51:16.724307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-897a42c7-69a0-4303-85ed-de7b9cd78a07 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ae05d51a1efc4c5398e9fa4ffeb7faa0', 'request_id': 'req-897a42c7-69a0-4303-85ed-de7b9cd78a07'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:16.793859 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-897a42c7-69a0-4303-85ed-de7b9cd78a07 admin admin] The token cache is being invalidate because project ae05d51a1efc4c5398e9fa4ffeb7faa0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:51:16.794105 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-897a42c7-69a0-4303-85ed-de7b9cd78a07 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:16.794768 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-897a42c7-69a0-4303-85ed-de7b9cd78a07 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:16.799873 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1262/2532] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:51:16 2022] DELETE /identity/v3/projects/ae05d51a1efc4c5398e9fa4ffeb7faa0 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:51:16.801580 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9a99ef-e3a0-475e-9e79-85b626f233fe None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:16.801870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9a99ef-e3a0-475e-9e79-85b626f233fe None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:16.802149 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9a99ef-e3a0-475e-9e79-85b626f233fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:16.832519 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-fb9a99ef-e3a0-475e-9e79-85b626f233fe None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:51:16.833764 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-fb9a99ef-e3a0-475e-9e79-85b626f233fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:16.902873 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1271/2533] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:51:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:51:16.911006 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb8d1ad-4426-4bd2-b283-9f43a7bda4a5 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:16.911325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb8d1ad-4426-4bd2-b283-9f43a7bda4a5 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:16.911596 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb8d1ad-4426-4bd2-b283-9f43a7bda4a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:16.932483 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-acb8d1ad-4426-4bd2-b283-9f43a7bda4a5 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:51:16.933685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-acb8d1ad-4426-4bd2-b283-9f43a7bda4a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:16.941552 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1263/2534] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:51:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:51:16.946021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9a99ef-e3a0-475e-9e79-85b626f233fe None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:16.955866 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9a99ef-e3a0-475e-9e79-85b626f233fe None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:16.961352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d643d5-f598-4d37-82ff-429b5c05cbdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:16.962238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d643d5-f598-4d37-82ff-429b5c05cbdf admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:16.962340 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d643d5-f598-4d37-82ff-429b5c05cbdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:16.962636 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d643d5-f598-4d37-82ff-429b5c05cbdf admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:16.963294 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d643d5-f598-4d37-82ff-429b5c05cbdf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:51:16.963652 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d643d5-f598-4d37-82ff-429b5c05cbdf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:16.965261 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d643d5-f598-4d37-82ff-429b5c05cbdf admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:16.978355 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1272/2535] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:51:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:51:16.983480 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb8d1ad-4426-4bd2-b283-9f43a7bda4a5 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:16.994790 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb8d1ad-4426-4bd2-b283-9f43a7bda4a5 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:16.999815 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f7b659-e38d-442f-8b7a-8bc349f515cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:17.000406 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f7b659-e38d-442f-8b7a-8bc349f515cd admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:17.000732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f7b659-e38d-442f-8b7a-8bc349f515cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:17.001051 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f7b659-e38d-442f-8b7a-8bc349f515cd admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:17.001669 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f7b659-e38d-442f-8b7a-8bc349f515cd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:17.003596 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f7b659-e38d-442f-8b7a-8bc349f515cd admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:17.093762 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1264/2536] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:51:16 2022] POST /identity/v3/projects => generated 379 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:51:17.097843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d643d5-f598-4d37-82ff-429b5c05cbdf admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:17.108596 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d643d5-f598-4d37-82ff-429b5c05cbdf admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:17.158040 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:51:17.165311 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7dd29d5-3df6-4953-8622-add06088b831 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:17.165912 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dd29d5-3df6-4953-8622-add06088b831 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:17.166531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dd29d5-3df6-4953-8622-add06088b831 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:17.166531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dd29d5-3df6-4953-8622-add06088b831 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:17.167104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7dd29d5-3df6-4953-8622-add06088b831 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:17.169035 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7dd29d5-3df6-4953-8622-add06088b831 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:17.284029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1273/2537] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:51:17 2022] POST /identity/v3/users => generated 290 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:51:17.289120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f7b659-e38d-442f-8b7a-8bc349f515cd admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:17.299541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f7b659-e38d-442f-8b7a-8bc349f515cd admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:17.304090 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4d77c5-a44c-452e-895b-c84c71f8a977 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:17.304679 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4d77c5-a44c-452e-895b-c84c71f8a977 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:17.305000 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4d77c5-a44c-452e-895b-c84c71f8a977 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:17.305281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4d77c5-a44c-452e-895b-c84c71f8a977 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:17.305926 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4d77c5-a44c-452e-895b-c84c71f8a977 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:17.307629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4d77c5-a44c-452e-895b-c84c71f8a977 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:17.317837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1265/2538] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:51:17 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:51:17.322539 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7dd29d5-3df6-4953-8622-add06088b831 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:17.332370 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7dd29d5-3df6-4953-8622-add06088b831 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:17.337726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f1f45d-aadb-4dbf-b349-9825872b25e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:17.338317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f1f45d-aadb-4dbf-b349-9825872b25e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:17.338610 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f1f45d-aadb-4dbf-b349-9825872b25e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:17.338981 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f1f45d-aadb-4dbf-b349-9825872b25e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:17.339630 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f1f45d-aadb-4dbf-b349-9825872b25e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:17.341226 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f1f45d-aadb-4dbf-b349-9825872b25e5 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:17.352427 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1274/2539] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:51:17 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:51:17.356471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4d77c5-a44c-452e-895b-c84c71f8a977 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:17.366730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4d77c5-a44c-452e-895b-c84c71f8a977 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:17.371405 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98237da2-0dd0-4865-b0f0-60b9e819ac6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:17.372058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98237da2-0dd0-4865-b0f0-60b9e819ac6b admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:17.372329 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98237da2-0dd0-4865-b0f0-60b9e819ac6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:17.372853 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98237da2-0dd0-4865-b0f0-60b9e819ac6b admin admin] PATH_INFO: `/v3/projects/f0d0c96bbb304882b1f2459361551a93/users/c1df4fe7054840feac0a648ce82eea1a/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:17.387874 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98237da2-0dd0-4865-b0f0-60b9e819ac6b admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0d0c96bbb304882b1f2459361551a93, user_id=c1df4fe7054840feac0a648ce82eea1a, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:17.390503 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98237da2-0dd0-4865-b0f0-60b9e819ac6b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:17.493455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1266/2540] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:51:17 2022] PUT /identity/v3/projects/f0d0c96bbb304882b1f2459361551a93/users/c1df4fe7054840feac0a648ce82eea1a/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:51:17.498761 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06989b81-a109-4e45-b96a-8123e844b15b None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:17.499049 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06989b81-a109-4e45-b96a-8123e844b15b None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:17.499321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06989b81-a109-4e45-b96a-8123e844b15b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:17.518862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-06989b81-a109-4e45-b96a-8123e844b15b None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:51:17.520096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-06989b81-a109-4e45-b96a-8123e844b15b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:17.578966 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1275/2541] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:51:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:51:17.589147 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98237da2-0dd0-4865-b0f0-60b9e819ac6b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:17.591026 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-98237da2-0dd0-4865-b0f0-60b9e819ac6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:17.647523 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98237da2-0dd0-4865-b0f0-60b9e819ac6b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:17.651691 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250d8af3-7d94-4edb-95d7-3af68f5bccd5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:17.652288 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250d8af3-7d94-4edb-95d7-3af68f5bccd5 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:17.652568 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250d8af3-7d94-4edb-95d7-3af68f5bccd5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:17.652841 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250d8af3-7d94-4edb-95d7-3af68f5bccd5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:17.661674 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250d8af3-7d94-4edb-95d7-3af68f5bccd5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:17.663134 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250d8af3-7d94-4edb-95d7-3af68f5bccd5 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:17.672445 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1267/2542] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:51:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:51:23.327513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06989b81-a109-4e45-b96a-8123e844b15b None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:23.329144 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-06989b81-a109-4e45-b96a-8123e844b15b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:23.492399 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06989b81-a109-4e45-b96a-8123e844b15b None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:23.506066 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1a8859-cd86-42a4-9397-c7af5ac996c3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:23.506715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1a8859-cd86-42a4-9397-c7af5ac996c3 service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:23.507170 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1a8859-cd86-42a4-9397-c7af5ac996c3 service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:23.507276 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1a8859-cd86-42a4-9397-c7af5ac996c3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:23.511433 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-8b1a8859-cd86-42a4-9397-c7af5ac996c3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:23.542775 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1a8859-cd86-42a4-9397-c7af5ac996c3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:23.544396 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1a8859-cd86-42a4-9397-c7af5ac996c3 service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:23.575462 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1276/2543] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:51:23 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 248 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:51:23.735665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cee702-ab5e-42c1-99bd-df949e92ec4e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:23.736802 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cee702-ab5e-42c1-99bd-df949e92ec4e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:23.737269 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cee702-ab5e-42c1-99bd-df949e92ec4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:23.789730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d5cee702-ab5e-42c1-99bd-df949e92ec4e None None] MFA Rules not processed for user `c1df4fe7054840feac0a648ce82eea1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:51:23.791037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d5cee702-ab5e-42c1-99bd-df949e92ec4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:23.892945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1268/2544] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:51:23 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:51:23.897243 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1a8859-cd86-42a4-9397-c7af5ac996c3 service nova] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:23.912476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1a8859-cd86-42a4-9397-c7af5ac996c3 service nova] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:23.919990 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5680e878-4439-4987-a8d6-f6d445d3bec4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:23.925725 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5680e878-4439-4987-a8d6-f6d445d3bec4 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:23.926017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5680e878-4439-4987-a8d6-f6d445d3bec4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:23.926365 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5680e878-4439-4987-a8d6-f6d445d3bec4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:23.927080 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5680e878-4439-4987-a8d6-f6d445d3bec4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:23.930169 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5680e878-4439-4987-a8d6-f6d445d3bec4 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:24.029918 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1277/2545] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:51:23 2022] POST /identity/v3/projects => generated 379 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:51:24.031244 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cee702-ab5e-42c1-99bd-df949e92ec4e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:24.043039 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cee702-ab5e-42c1-99bd-df949e92ec4e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:24.056223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:51:24.105867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bacadbe-611f-4741-92bd-17b3a60617db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:24.105867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bacadbe-611f-4741-92bd-17b3a60617db admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:24.105867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bacadbe-611f-4741-92bd-17b3a60617db admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:24.106228 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bacadbe-611f-4741-92bd-17b3a60617db admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:24.106785 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bacadbe-611f-4741-92bd-17b3a60617db admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:24.108877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bacadbe-611f-4741-92bd-17b3a60617db admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:24.196953 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1269/2546] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:51:24 2022] POST /identity/v3/users => generated 296 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:51:24.200989 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5680e878-4439-4987-a8d6-f6d445d3bec4 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:24.212174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5680e878-4439-4987-a8d6-f6d445d3bec4 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:24.216850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce468a70-9ef7-4c50-955d-2729831a0976 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:24.217458 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce468a70-9ef7-4c50-955d-2729831a0976 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:24.217740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce468a70-9ef7-4c50-955d-2729831a0976 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:24.218640 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce468a70-9ef7-4c50-955d-2729831a0976 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:24.219306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce468a70-9ef7-4c50-955d-2729831a0976 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:24.220911 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce468a70-9ef7-4c50-955d-2729831a0976 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:24.231388 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1278/2547] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:51:24 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:51:24.235418 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bacadbe-611f-4741-92bd-17b3a60617db admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:24.245444 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bacadbe-611f-4741-92bd-17b3a60617db admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:24.250360 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d767177-c7be-4573-ac02-b15263fe1a57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:24.251056 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d767177-c7be-4573-ac02-b15263fe1a57 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:24.251340 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d767177-c7be-4573-ac02-b15263fe1a57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:24.251620 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d767177-c7be-4573-ac02-b15263fe1a57 admin admin] PATH_INFO: `/v3/projects/8003aa7a2df842bb92467d64e17bab8d/users/8a29648893a64ebb955d00833bc398e9/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:24.266826 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d767177-c7be-4573-ac02-b15263fe1a57 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8003aa7a2df842bb92467d64e17bab8d, user_id=8a29648893a64ebb955d00833bc398e9, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:24.269519 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d767177-c7be-4573-ac02-b15263fe1a57 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:24.341558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1270/2548] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:51:24 2022] PUT /identity/v3/projects/8003aa7a2df842bb92467d64e17bab8d/users/8a29648893a64ebb955d00833bc398e9/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:51:24.348875 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce468a70-9ef7-4c50-955d-2729831a0976 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:24.360254 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce468a70-9ef7-4c50-955d-2729831a0976 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:24.389690 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:51:24.424120 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65cda1b-b221-455c-a215-cac222aa1d70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:24.424731 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65cda1b-b221-455c-a215-cac222aa1d70 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:24.425035 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65cda1b-b221-455c-a215-cac222aa1d70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:24.425311 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65cda1b-b221-455c-a215-cac222aa1d70 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:24.425961 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e65cda1b-b221-455c-a215-cac222aa1d70 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:24.427605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e65cda1b-b221-455c-a215-cac222aa1d70 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:24.441757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1279/2549] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:51:24 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:51:24.442663 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d767177-c7be-4573-ac02-b15263fe1a57 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:24.453967 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d767177-c7be-4573-ac02-b15263fe1a57 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:24.463488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b1877d-4817-4632-85a2-687255504cb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:24.464173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b1877d-4817-4632-85a2-687255504cb1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:24.464453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b1877d-4817-4632-85a2-687255504cb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:24.464727 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b1877d-4817-4632-85a2-687255504cb1 admin admin] PATH_INFO: `/v3/projects/8003aa7a2df842bb92467d64e17bab8d/users/8a29648893a64ebb955d00833bc398e9/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:24.466792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b1877d-4817-4632-85a2-687255504cb1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8003aa7a2df842bb92467d64e17bab8d, user_id=8a29648893a64ebb955d00833bc398e9, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:24.469468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b1877d-4817-4632-85a2-687255504cb1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:24.482027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1271/2550] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:51:24 2022] PUT /identity/v3/projects/8003aa7a2df842bb92467d64e17bab8d/users/8a29648893a64ebb955d00833bc398e9/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:51:26.167258 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65cda1b-b221-455c-a215-cac222aa1d70 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:26.289591 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65cda1b-b221-455c-a215-cac222aa1d70 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:26.369479 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487a0f47-311a-4c82-87c8-4ce1487165aa service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:26.370973 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487a0f47-311a-4c82-87c8-4ce1487165aa service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:26.371368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487a0f47-311a-4c82-87c8-4ce1487165aa service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:26.371743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487a0f47-311a-4c82-87c8-4ce1487165aa service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:26.375427 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-487a0f47-311a-4c82-87c8-4ce1487165aa service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:26.480153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487a0f47-311a-4c82-87c8-4ce1487165aa service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:26.482040 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487a0f47-311a-4c82-87c8-4ce1487165aa service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:26.486523 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:51:26.496564 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1280/2551] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:51:26 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:51:31.027275 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcef115-200c-42b7-9e52-e99b3f9fcb3d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:31.027275 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcef115-200c-42b7-9e52-e99b3f9fcb3d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:31.027275 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcef115-200c-42b7-9e52-e99b3f9fcb3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:31.061562 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-0bcef115-200c-42b7-9e52-e99b3f9fcb3d None None] MFA Rules not processed for user `8a29648893a64ebb955d00833bc398e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:51:31.062937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0bcef115-200c-42b7-9e52-e99b3f9fcb3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:31.145472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1272/2552] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:51:31 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:51:31.165573 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487a0f47-311a-4c82-87c8-4ce1487165aa service nova] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:31.184756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487a0f47-311a-4c82-87c8-4ce1487165aa service nova] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:31.193815 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731ee40f-65cc-419a-befa-7d01887901bf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:31.196466 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731ee40f-65cc-419a-befa-7d01887901bf service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:31.196466 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731ee40f-65cc-419a-befa-7d01887901bf service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:31.196466 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731ee40f-65cc-419a-befa-7d01887901bf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:31.289482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731ee40f-65cc-419a-befa-7d01887901bf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:31.291222 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731ee40f-65cc-419a-befa-7d01887901bf service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:31.301157 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1281/2553] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:51:31 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:51:41.141793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=70,keystone:INSERT=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:51:41.296718 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=29 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:51:44.448656 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcef115-200c-42b7-9e52-e99b3f9fcb3d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:44.462826 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcef115-200c-42b7-9e52-e99b3f9fcb3d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:44.468870 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1273e2-f3de-466c-8f9a-023be912836e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:44.469555 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1273e2-f3de-466c-8f9a-023be912836e service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:44.469839 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1273e2-f3de-466c-8f9a-023be912836e service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:44.470222 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1273e2-f3de-466c-8f9a-023be912836e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:44.471936 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5f1273e2-f3de-466c-8f9a-023be912836e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:44.485400 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1273e2-f3de-466c-8f9a-023be912836e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:44.487095 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1273e2-f3de-466c-8f9a-023be912836e service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:44.497795 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1273/2554] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:51:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:51:45.942581 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731ee40f-65cc-419a-befa-7d01887901bf service nova] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:45.996089 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731ee40f-65cc-419a-befa-7d01887901bf service nova] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:45.999960 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0055c752-f2d4-41ed-87ad-1c9653a5e182 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:46.000585 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0055c752-f2d4-41ed-87ad-1c9653a5e182 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:46.000890 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0055c752-f2d4-41ed-87ad-1c9653a5e182 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:46.001157 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0055c752-f2d4-41ed-87ad-1c9653a5e182 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:46.002893 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0055c752-f2d4-41ed-87ad-1c9653a5e182 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:46.063635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0055c752-f2d4-41ed-87ad-1c9653a5e182 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:46.065064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0055c752-f2d4-41ed-87ad-1c9653a5e182 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:46.074487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1282/2555] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:51:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:51:49.993291 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1273e2-f3de-466c-8f9a-023be912836e service nova] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:49.994787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5f1273e2-f3de-466c-8f9a-023be912836e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:50.008919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1273e2-f3de-466c-8f9a-023be912836e service nova] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:50.019129 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a05000-dc78-4126-9431-143717925e0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:50.019129 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a05000-dc78-4126-9431-143717925e0e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:50.019129 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a05000-dc78-4126-9431-143717925e0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:50.019129 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a05000-dc78-4126-9431-143717925e0e admin admin] PATH_INFO: `/v3/users/7ad21da05fc246d489b0afa641f20216` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:50.019522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a05000-dc78-4126-9431-143717925e0e admin admin] RBAC: Authorizing `identity:delete_user(user_id=7ad21da05fc246d489b0afa641f20216)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:50.019618 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a05000-dc78-4126-9431-143717925e0e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:50.189744 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-05a05000-dc78-4126-9431-143717925e0e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7ad21da05fc246d489b0afa641f20216'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:50.190326 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-05a05000-dc78-4126-9431-143717925e0e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7ad21da05fc246d489b0afa641f20216'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:50.191128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-05a05000-dc78-4126-9431-143717925e0e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7ad21da05fc246d489b0afa641f20216'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:50.198453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-05a05000-dc78-4126-9431-143717925e0e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7ad21da05fc246d489b0afa641f20216'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:50.275145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-05a05000-dc78-4126-9431-143717925e0e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7ad21da05fc246d489b0afa641f20216'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:50.293405 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1274/2556] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:51:49 2022] DELETE /identity/v3/users/7ad21da05fc246d489b0afa641f20216 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:51:50.780440 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0055c752-f2d4-41ed-87ad-1c9653a5e182 service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:50.790813 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0055c752-f2d4-41ed-87ad-1c9653a5e182 service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:50.850186 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869fdbbc-fc71-4fe5-b2db-42937f93815c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:50.850863 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869fdbbc-fc71-4fe5-b2db-42937f93815c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:50.851148 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869fdbbc-fc71-4fe5-b2db-42937f93815c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:50.851422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869fdbbc-fc71-4fe5-b2db-42937f93815c admin admin] PATH_INFO: `/v3/projects/f1c0948ddb014e6f93d6069bb5203890` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:50.852530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869fdbbc-fc71-4fe5-b2db-42937f93815c admin admin] RBAC: Authorizing `identity:delete_project(project_id=f1c0948ddb014e6f93d6069bb5203890)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:50.854472 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869fdbbc-fc71-4fe5-b2db-42937f93815c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:50.967755 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-869fdbbc-fc71-4fe5-b2db-42937f93815c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f1c0948ddb014e6f93d6069bb5203890', 'request_id': 'req-869fdbbc-fc71-4fe5-b2db-42937f93815c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:50.969720 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:51:50.973725 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-869fdbbc-fc71-4fe5-b2db-42937f93815c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f1c0948ddb014e6f93d6069bb5203890', 'request_id': 'req-869fdbbc-fc71-4fe5-b2db-42937f93815c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:50.980605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-869fdbbc-fc71-4fe5-b2db-42937f93815c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f1c0948ddb014e6f93d6069bb5203890', 'request_id': 'req-869fdbbc-fc71-4fe5-b2db-42937f93815c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:51.066289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-869fdbbc-fc71-4fe5-b2db-42937f93815c admin admin] The token cache is being invalidate because project f1c0948ddb014e6f93d6069bb5203890 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:51:51.066484 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-869fdbbc-fc71-4fe5-b2db-42937f93815c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:51.067135 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-869fdbbc-fc71-4fe5-b2db-42937f93815c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:51:51.068660 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1283/2557] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:51:50 2022] DELETE /identity/v3/projects/f1c0948ddb014e6f93d6069bb5203890 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:51:51.073878 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a56ea1-3a47-46da-9c9e-89c8e82631c9 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:51.074364 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a56ea1-3a47-46da-9c9e-89c8e82631c9 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:51.074471 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a56ea1-3a47-46da-9c9e-89c8e82631c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:51.093770 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-a7a56ea1-3a47-46da-9c9e-89c8e82631c9 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:51:51.095016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a7a56ea1-3a47-46da-9c9e-89c8e82631c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:51.153770 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1275/2558] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:51:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:51:51.161584 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40a60e1-e0ab-48ac-80cf-3f4ab6ee913b None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:51.161877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40a60e1-e0ab-48ac-80cf-3f4ab6ee913b None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:51.162156 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40a60e1-e0ab-48ac-80cf-3f4ab6ee913b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:51.182112 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-f40a60e1-e0ab-48ac-80cf-3f4ab6ee913b None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:51:51.184320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-f40a60e1-e0ab-48ac-80cf-3f4ab6ee913b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:51.192585 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1284/2559] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:51:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:51:51.196604 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a56ea1-3a47-46da-9c9e-89c8e82631c9 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:51.206719 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a56ea1-3a47-46da-9c9e-89c8e82631c9 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:51.211422 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5d9b38-229a-4d82-a55c-781feb0c67eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:51.212029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5d9b38-229a-4d82-a55c-781feb0c67eb admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:51.212305 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5d9b38-229a-4d82-a55c-781feb0c67eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:51.212582 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5d9b38-229a-4d82-a55c-781feb0c67eb admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:51.213224 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5d9b38-229a-4d82-a55c-781feb0c67eb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:51:51.213529 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5d9b38-229a-4d82-a55c-781feb0c67eb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:51.215290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5d9b38-229a-4d82-a55c-781feb0c67eb admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:51.227444 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1276/2560] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:51:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 30 09:51:51.232884 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40a60e1-e0ab-48ac-80cf-3f4ab6ee913b None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:51.242264 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40a60e1-e0ab-48ac-80cf-3f4ab6ee913b None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:51.247690 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dcad29-0001-4efa-81f3-a2df06d49171 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:51.248627 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dcad29-0001-4efa-81f3-a2df06d49171 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:51.249164 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dcad29-0001-4efa-81f3-a2df06d49171 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:51.249441 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dcad29-0001-4efa-81f3-a2df06d49171 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:51.250447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0dcad29-0001-4efa-81f3-a2df06d49171 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:51.252103 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0dcad29-0001-4efa-81f3-a2df06d49171 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:51.374753 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1285/2561] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:51:51 2022] POST /identity/v3/projects => generated 389 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:51:51.378847 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5d9b38-229a-4d82-a55c-781feb0c67eb admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:51.389020 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5d9b38-229a-4d82-a55c-781feb0c67eb admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:51.432192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:51:51.444616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1877391b-8ecf-4972-bb9d-a8f962eaf284 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:51.445243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1877391b-8ecf-4972-bb9d-a8f962eaf284 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:51.445525 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1877391b-8ecf-4972-bb9d-a8f962eaf284 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:51.448594 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1877391b-8ecf-4972-bb9d-a8f962eaf284 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:51.448594 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1877391b-8ecf-4972-bb9d-a8f962eaf284 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:51.448594 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1877391b-8ecf-4972-bb9d-a8f962eaf284 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:51.538057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1277/2562] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:51:51 2022] POST /identity/v3/users => generated 295 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:51:51.542920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dcad29-0001-4efa-81f3-a2df06d49171 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:51.554002 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dcad29-0001-4efa-81f3-a2df06d49171 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:51.558848 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939e0aca-4b1a-4f4d-a486-9433b2f9dcc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:51.559447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939e0aca-4b1a-4f4d-a486-9433b2f9dcc7 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:51.559752 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939e0aca-4b1a-4f4d-a486-9433b2f9dcc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:51.560050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939e0aca-4b1a-4f4d-a486-9433b2f9dcc7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:51.560704 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939e0aca-4b1a-4f4d-a486-9433b2f9dcc7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:51.562288 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939e0aca-4b1a-4f4d-a486-9433b2f9dcc7 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:51.572307 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1286/2563] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:51:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:51:51.576360 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1877391b-8ecf-4972-bb9d-a8f962eaf284 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:51.587083 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1877391b-8ecf-4972-bb9d-a8f962eaf284 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:51.592343 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a41ad0-0155-4d22-bba1-a633f20dacdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:51.592935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a41ad0-0155-4d22-bba1-a633f20dacdb admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:51.593217 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a41ad0-0155-4d22-bba1-a633f20dacdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:51.593490 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a41ad0-0155-4d22-bba1-a633f20dacdb admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:51.594126 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a41ad0-0155-4d22-bba1-a633f20dacdb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:51.595756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a41ad0-0155-4d22-bba1-a633f20dacdb admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:51.607292 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1278/2564] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:51:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:51:51.611589 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939e0aca-4b1a-4f4d-a486-9433b2f9dcc7 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:51.622360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939e0aca-4b1a-4f4d-a486-9433b2f9dcc7 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:51.627323 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d428f2-b3ab-4b3f-a866-4577923f9fea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:51.627903 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d428f2-b3ab-4b3f-a866-4577923f9fea admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:51.628184 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d428f2-b3ab-4b3f-a866-4577923f9fea admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:51.628460 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d428f2-b3ab-4b3f-a866-4577923f9fea admin admin] PATH_INFO: `/v3/projects/1142014cc9284cf1833e5bfbcdb41599/users/21d5e60badb846fb840a65f443728a9b/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:51.642796 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d428f2-b3ab-4b3f-a866-4577923f9fea admin admin] RBAC: Authorizing `identity:create_grant(project_id=1142014cc9284cf1833e5bfbcdb41599, user_id=21d5e60badb846fb840a65f443728a9b, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:51.645453 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d428f2-b3ab-4b3f-a866-4577923f9fea admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:51.692617 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1287/2565] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:51:51 2022] PUT /identity/v3/projects/1142014cc9284cf1833e5bfbcdb41599/users/21d5e60badb846fb840a65f443728a9b/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:51:51.697877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf81b840-da6a-4874-8842-1cca3a1f7eb6 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:51.698179 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf81b840-da6a-4874-8842-1cca3a1f7eb6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:51.698443 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf81b840-da6a-4874-8842-1cca3a1f7eb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:51.717983 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-cf81b840-da6a-4874-8842-1cca3a1f7eb6 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:51:51.719243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-cf81b840-da6a-4874-8842-1cca3a1f7eb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:51.789808 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1279/2566] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:51:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:51:51.799485 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d428f2-b3ab-4b3f-a866-4577923f9fea admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:51.801219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-54d428f2-b3ab-4b3f-a866-4577923f9fea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:51.878901 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d428f2-b3ab-4b3f-a866-4577923f9fea admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:51.879324 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71def462-1030-4ab7-b936-3fa506dd57ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:51.880463 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71def462-1030-4ab7-b936-3fa506dd57ac service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:51.880772 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71def462-1030-4ab7-b936-3fa506dd57ac service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:51.881050 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71def462-1030-4ab7-b936-3fa506dd57ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:51.890968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71def462-1030-4ab7-b936-3fa506dd57ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:51.892453 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71def462-1030-4ab7-b936-3fa506dd57ac service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:51.902937 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1288/2567] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:51:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:51:52.999583 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85ecb95-555e-458d-a7d9-2e6b9d4bba43 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:53.000408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85ecb95-555e-458d-a7d9-2e6b9d4bba43 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:53.000408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85ecb95-555e-458d-a7d9-2e6b9d4bba43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:53.019355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d85ecb95-555e-458d-a7d9-2e6b9d4bba43 None None] MFA Rules not processed for user `c1df4fe7054840feac0a648ce82eea1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:51:53.020543 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d85ecb95-555e-458d-a7d9-2e6b9d4bba43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:53.081717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1280/2568] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:51:52 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:51:53.462246 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71def462-1030-4ab7-b936-3fa506dd57ac service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:53.471752 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71def462-1030-4ab7-b936-3fa506dd57ac service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:53.475964 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efb2de8-9e9c-479d-bf1d-254e2ef9bfd7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:53.476713 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efb2de8-9e9c-479d-bf1d-254e2ef9bfd7 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:53.477003 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efb2de8-9e9c-479d-bf1d-254e2ef9bfd7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:53.477305 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efb2de8-9e9c-479d-bf1d-254e2ef9bfd7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:53.479068 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-8efb2de8-9e9c-479d-bf1d-254e2ef9bfd7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:53.490326 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efb2de8-9e9c-479d-bf1d-254e2ef9bfd7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:53.491824 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efb2de8-9e9c-479d-bf1d-254e2ef9bfd7 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:53.500687 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1289/2569] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:51:53 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:51:54.402338 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85ecb95-555e-458d-a7d9-2e6b9d4bba43 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:54.404188 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d85ecb95-555e-458d-a7d9-2e6b9d4bba43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:54.502361 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85ecb95-555e-458d-a7d9-2e6b9d4bba43 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:54.509775 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9f60af-16fe-4e8e-b9ab-f894ca465ffe service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '49a3011c103b4e48904f876f81d27fba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:54.510414 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9f60af-16fe-4e8e-b9ab-f894ca465ffe service placement] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:54.510761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9f60af-16fe-4e8e-b9ab-f894ca465ffe service placement] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:54.511043 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9f60af-16fe-4e8e-b9ab-f894ca465ffe service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:54.513098 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9a9f60af-16fe-4e8e-b9ab-f894ca465ffe service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:54.532400 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9f60af-16fe-4e8e-b9ab-f894ca465ffe service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:54.534015 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9f60af-16fe-4e8e-b9ab-f894ca465ffe service placement] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:54.541604 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:51:54.548996 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1281/2570] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:51:54 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:51:56.069981 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf148d2-dd62-46e3-8331-531a2e1450c9 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:56.069981 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf148d2-dd62-46e3-8331-531a2e1450c9 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:56.069981 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf148d2-dd62-46e3-8331-531a2e1450c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:56.090676 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-7cf148d2-dd62-46e3-8331-531a2e1450c9 None None] MFA Rules not processed for user `8c4ced79c3ff4842b30b472acb555cd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:51:56.092178 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7cf148d2-dd62-46e3-8331-531a2e1450c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:56.155319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1290/2571] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:51:56 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:51:56.701261 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9f60af-16fe-4e8e-b9ab-f894ca465ffe service placement] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:56.715739 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9f60af-16fe-4e8e-b9ab-f894ca465ffe service placement] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:56.721763 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd3ec59-d63a-4f51-afdc-b4e0e1012d99 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:56.723558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd3ec59-d63a-4f51-afdc-b4e0e1012d99 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:56.724271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd3ec59-d63a-4f51-afdc-b4e0e1012d99 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:56.724618 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd3ec59-d63a-4f51-afdc-b4e0e1012d99 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:56.726532 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2fd3ec59-d63a-4f51-afdc-b4e0e1012d99 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:56.741921 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd3ec59-d63a-4f51-afdc-b4e0e1012d99 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:56.743609 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd3ec59-d63a-4f51-afdc-b4e0e1012d99 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:56.753581 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1282/2572] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:51:56 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:51:58.427510 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc2f0cb-8c0f-4adc-8c57-92337f745606 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:58.427996 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc2f0cb-8c0f-4adc-8c57-92337f745606 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:58.428443 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc2f0cb-8c0f-4adc-8c57-92337f745606 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:58.471708 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-8cc2f0cb-8c0f-4adc-8c57-92337f745606 None None] MFA Rules not processed for user `8f8db95b40ca4350b4f86ce9f1593194`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:51:58.479141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-8cc2f0cb-8c0f-4adc-8c57-92337f745606 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:58.490722 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:51:58.593459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1291/2573] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:51:58 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:51:58.662841 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1331d392-0c37-46f0-bdcf-fe85a811e43c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:58.663786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1331d392-0c37-46f0-bdcf-fe85a811e43c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:58.664223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1331d392-0c37-46f0-bdcf-fe85a811e43c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:58.696372 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-1331d392-0c37-46f0-bdcf-fe85a811e43c None None] MFA Rules not processed for user `21d5e60badb846fb840a65f443728a9b`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:51:58.698001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1331d392-0c37-46f0-bdcf-fe85a811e43c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:58.788541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1283/2574] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:51:58 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:51:58.803417 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc2f0cb-8c0f-4adc-8c57-92337f745606 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:51:58.804914 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-8cc2f0cb-8c0f-4adc-8c57-92337f745606 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:51:58.907550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc2f0cb-8c0f-4adc-8c57-92337f745606 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:51:58.914187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704e3e69-9251-4e5c-97d6-e4c5e4fcb9c5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:51:58.914847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704e3e69-9251-4e5c-97d6-e4c5e4fcb9c5 service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:51:58.915132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704e3e69-9251-4e5c-97d6-e4c5e4fcb9c5 service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:51:58.915408 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704e3e69-9251-4e5c-97d6-e4c5e4fcb9c5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:51:58.924896 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704e3e69-9251-4e5c-97d6-e4c5e4fcb9c5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:51:58.926449 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704e3e69-9251-4e5c-97d6-e4c5e4fcb9c5 service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:51:58.950722 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1292/2575] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:51:58 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:52:01.612218 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1331d392-0c37-46f0-bdcf-fe85a811e43c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:52:01.631487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1331d392-0c37-46f0-bdcf-fe85a811e43c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:52:01.647877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d741a9d-b7a2-400f-980e-22e7ff7e8f19 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:52:01.648508 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d741a9d-b7a2-400f-980e-22e7ff7e8f19 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:52:01.648893 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d741a9d-b7a2-400f-980e-22e7ff7e8f19 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:52:01.649038 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d741a9d-b7a2-400f-980e-22e7ff7e8f19 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:52:01.656899 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9d741a9d-b7a2-400f-980e-22e7ff7e8f19 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:52:01.673850 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d741a9d-b7a2-400f-980e-22e7ff7e8f19 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:52:01.675907 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d741a9d-b7a2-400f-980e-22e7ff7e8f19 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:52:01.697877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1284/2576] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:52:01 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:52:03.101401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ba775e-c398-43ea-94b3-62722adf0e80 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:52:03.102378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ba775e-c398-43ea-94b3-62722adf0e80 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:52:03.102973 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ba775e-c398-43ea-94b3-62722adf0e80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:52:03.124833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-f7ba775e-c398-43ea-94b3-62722adf0e80 None None] MFA Rules not processed for user `21d5e60badb846fb840a65f443728a9b`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:52:03.126134 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-f7ba775e-c398-43ea-94b3-62722adf0e80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:52:03.135263 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1293/2577] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:52:03 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:52:10.638195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d741a9d-b7a2-400f-980e-22e7ff7e8f19 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:52:10.639581 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9d741a9d-b7a2-400f-980e-22e7ff7e8f19 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:52:10.659411 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d741a9d-b7a2-400f-980e-22e7ff7e8f19 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:52:10.665422 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa028aae-9601-4c8c-9230-ba71ac1f6367 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:52:10.666364 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa028aae-9601-4c8c-9230-ba71ac1f6367 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:52:10.667359 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa028aae-9601-4c8c-9230-ba71ac1f6367 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:52:10.672853 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa028aae-9601-4c8c-9230-ba71ac1f6367 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:52:10.672853 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-aa028aae-9601-4c8c-9230-ba71ac1f6367 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:52:10.686087 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa028aae-9601-4c8c-9230-ba71ac1f6367 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:52:10.688961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa028aae-9601-4c8c-9230-ba71ac1f6367 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:52:10.710800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1285/2578] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:52:10 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 73 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:52:13.118780 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=57 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:52:14.329618 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6cd80f-e6ed-4f90-96fb-bad06fabd2cd None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:52:14.330306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6cd80f-e6ed-4f90-96fb-bad06fabd2cd None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:52:14.330306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6cd80f-e6ed-4f90-96fb-bad06fabd2cd None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:52:14.331629 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1294/2579] 10.222.0.39 () {54 vars in 921 bytes} [Fri Sep 30 09:52:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:52:14.338542 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4aa559-4c0e-4d6a-8fe0-84e98fc135ac None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:52:14.339536 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4aa559-4c0e-4d6a-8fe0-84e98fc135ac None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:52:14.339642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4aa559-4c0e-4d6a-8fe0-84e98fc135ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:52:14.365029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-4e4aa559-4c0e-4d6a-8fe0-84e98fc135ac None None] MFA Rules not processed for user `9266babd21414ae9998716e5a0478e9d`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:52:14.367381 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4e4aa559-4c0e-4d6a-8fe0-84e98fc135ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:52:14.376802 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1286/2580] 10.222.0.39 () {58 vars in 1018 bytes} [Fri Sep 30 09:52:14 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:52:14.389989 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6cd80f-e6ed-4f90-96fb-bad06fabd2cd None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:52:14.392034 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7d6cd80f-e6ed-4f90-96fb-bad06fabd2cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:52:14.411281 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6cd80f-e6ed-4f90-96fb-bad06fabd2cd None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:52:14.415252 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308ab88d-4971-43fc-9948-d1bf5ac727e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:52:14.415935 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308ab88d-4971-43fc-9948-d1bf5ac727e5 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:52:14.416289 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308ab88d-4971-43fc-9948-d1bf5ac727e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:52:14.416693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308ab88d-4971-43fc-9948-d1bf5ac727e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:52:14.426858 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308ab88d-4971-43fc-9948-d1bf5ac727e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:52:14.428811 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308ab88d-4971-43fc-9948-d1bf5ac727e5 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:52:14.453013 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1295/2581] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:52:14 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:52:14.549419 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8851eff-1e60-400f-8fec-2a86a182624a None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:52:14.549904 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8851eff-1e60-400f-8fec-2a86a182624a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:52:14.550350 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8851eff-1e60-400f-8fec-2a86a182624a None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:52:14.552037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1287/2582] 10.222.0.39 () {54 vars in 921 bytes} [Fri Sep 30 09:52:14 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:52:14.560387 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1b16b1-ed18-4b28-850c-ab5e9dc7560e None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:52:14.566114 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1b16b1-ed18-4b28-850c-ab5e9dc7560e None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:52:14.566114 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1b16b1-ed18-4b28-850c-ab5e9dc7560e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:52:14.591775 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-ef1b16b1-ed18-4b28-850c-ab5e9dc7560e None None] MFA Rules not processed for user `7ff68fb87700423ab53c90520a6bd3b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:52:14.593082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ef1b16b1-ed18-4b28-850c-ab5e9dc7560e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:52:14.603910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1296/2583] 10.222.0.39 () {58 vars in 1018 bytes} [Fri Sep 30 09:52:14 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:52:14.617711 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8851eff-1e60-400f-8fec-2a86a182624a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:52:14.630893 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8851eff-1e60-400f-8fec-2a86a182624a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:52:14.636955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9ea8f6-b901-46d1-957e-ea40210b5dfd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:52:14.637620 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9ea8f6-b901-46d1-957e-ea40210b5dfd service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:52:14.638200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9ea8f6-b901-46d1-957e-ea40210b5dfd service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:52:14.638481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9ea8f6-b901-46d1-957e-ea40210b5dfd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:52:14.650137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9ea8f6-b901-46d1-957e-ea40210b5dfd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:52:14.651647 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9ea8f6-b901-46d1-957e-ea40210b5dfd service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:52:14.664251 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1288/2584] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:52:14 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:52:20.519231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1b16b1-ed18-4b28-850c-ab5e9dc7560e None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:52:20.529156 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1b16b1-ed18-4b28-850c-ab5e9dc7560e None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:52:20.532921 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c68592d-41b4-4ed3-934b-5d83bade388b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:52:20.533548 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c68592d-41b4-4ed3-934b-5d83bade388b service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:52:20.533828 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c68592d-41b4-4ed3-934b-5d83bade388b service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:52:20.534096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c68592d-41b4-4ed3-934b-5d83bade388b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:52:20.535900 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7c68592d-41b4-4ed3-934b-5d83bade388b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:52:20.600286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c68592d-41b4-4ed3-934b-5d83bade388b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:52:20.601734 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c68592d-41b4-4ed3-934b-5d83bade388b service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:52:20.611707 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1297/2585] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:52:20 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:52:24.659287 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=59 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:52:30.607332 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=39 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:14.855625 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9ea8f6-b901-46d1-957e-ea40210b5dfd service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:14.899017 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9ea8f6-b901-46d1-957e-ea40210b5dfd service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:14.904156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbc3af1-4b1c-4edd-857a-660490f411ed service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '49a3011c103b4e48904f876f81d27fba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:14.904784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbc3af1-4b1c-4edd-857a-660490f411ed service placement] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:14.905065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbc3af1-4b1c-4edd-857a-660490f411ed service placement] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:14.905341 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbc3af1-4b1c-4edd-857a-660490f411ed service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:14.907145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6bbc3af1-4b1c-4edd-857a-660490f411ed service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:14.938547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbc3af1-4b1c-4edd-857a-660490f411ed service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:14.940148 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbc3af1-4b1c-4edd-857a-660490f411ed service placement] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:14.950109 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1289/2586] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:53:14 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:53:14.981745 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c68592d-41b4-4ed3-934b-5d83bade388b service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:14.983798 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7c68592d-41b4-4ed3-934b-5d83bade388b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:15.079674 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c68592d-41b4-4ed3-934b-5d83bade388b service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:15.084820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cb2175-c326-4a07-a66f-1147d26b4984 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:15.085439 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cb2175-c326-4a07-a66f-1147d26b4984 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:15.085716 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cb2175-c326-4a07-a66f-1147d26b4984 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:15.085987 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cb2175-c326-4a07-a66f-1147d26b4984 admin admin] PATH_INFO: `/v3/users/8f8db95b40ca4350b4f86ce9f1593194` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:15.087185 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cb2175-c326-4a07-a66f-1147d26b4984 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8f8db95b40ca4350b4f86ce9f1593194)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:15.089090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cb2175-c326-4a07-a66f-1147d26b4984 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:15.295284 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-85cb2175-c326-4a07-a66f-1147d26b4984 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8f8db95b40ca4350b4f86ce9f1593194'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:15.308310 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-85cb2175-c326-4a07-a66f-1147d26b4984 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8f8db95b40ca4350b4f86ce9f1593194'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:15.309021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-85cb2175-c326-4a07-a66f-1147d26b4984 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8f8db95b40ca4350b4f86ce9f1593194'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:15.395668 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-85cb2175-c326-4a07-a66f-1147d26b4984 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8f8db95b40ca4350b4f86ce9f1593194'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:15.402942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-85cb2175-c326-4a07-a66f-1147d26b4984 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8f8db95b40ca4350b4f86ce9f1593194'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:15.408857 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1298/2587] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:53:14 2022] DELETE /identity/v3/users/8f8db95b40ca4350b4f86ce9f1593194 => generated 0 bytes in 428 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:15.669538 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbc3af1-4b1c-4edd-857a-660490f411ed service placement] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:15.682496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbc3af1-4b1c-4edd-857a-660490f411ed service placement] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:15.738957 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1673aaf0-332d-4b15-b41b-fe88f5b7af70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:15.739589 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1673aaf0-332d-4b15-b41b-fe88f5b7af70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:15.739868 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1673aaf0-332d-4b15-b41b-fe88f5b7af70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:15.740140 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1673aaf0-332d-4b15-b41b-fe88f5b7af70 admin admin] PATH_INFO: `/v3/projects/b3db64185ae94cefa4efa6f3e03c2f7e` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:15.741242 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1673aaf0-332d-4b15-b41b-fe88f5b7af70 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b3db64185ae94cefa4efa6f3e03c2f7e)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:15.743193 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1673aaf0-332d-4b15-b41b-fe88f5b7af70 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:15.834859 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1673aaf0-332d-4b15-b41b-fe88f5b7af70 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b3db64185ae94cefa4efa6f3e03c2f7e', 'request_id': 'req-1673aaf0-332d-4b15-b41b-fe88f5b7af70'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:15.840873 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1673aaf0-332d-4b15-b41b-fe88f5b7af70 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b3db64185ae94cefa4efa6f3e03c2f7e', 'request_id': 'req-1673aaf0-332d-4b15-b41b-fe88f5b7af70'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:15.847269 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1673aaf0-332d-4b15-b41b-fe88f5b7af70 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b3db64185ae94cefa4efa6f3e03c2f7e', 'request_id': 'req-1673aaf0-332d-4b15-b41b-fe88f5b7af70'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:15.913249 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1673aaf0-332d-4b15-b41b-fe88f5b7af70 admin admin] The token cache is being invalidate because project b3db64185ae94cefa4efa6f3e03c2f7e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:53:15.913886 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1673aaf0-332d-4b15-b41b-fe88f5b7af70 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:15.914202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1673aaf0-332d-4b15-b41b-fe88f5b7af70 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:15.915766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1290/2588] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:53:15 2022] DELETE /identity/v3/projects/b3db64185ae94cefa4efa6f3e03c2f7e => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:25.405386 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=57,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:25.910013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=79,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:26.126644 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cb2175-c326-4a07-a66f-1147d26b4984 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:26.129014 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-85cb2175-c326-4a07-a66f-1147d26b4984 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:26.207057 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cb2175-c326-4a07-a66f-1147d26b4984 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:26.254836 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4413e24c-72da-49fb-b167-2c5552b49ba8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:26.255442 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4413e24c-72da-49fb-b167-2c5552b49ba8 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:26.255719 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4413e24c-72da-49fb-b167-2c5552b49ba8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:26.255989 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4413e24c-72da-49fb-b167-2c5552b49ba8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:26.257710 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-4413e24c-72da-49fb-b167-2c5552b49ba8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:26.317218 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4413e24c-72da-49fb-b167-2c5552b49ba8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:26.318618 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4413e24c-72da-49fb-b167-2c5552b49ba8 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:26.327961 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1299/2589] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:53:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:53:30.229733 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1673aaf0-332d-4b15-b41b-fe88f5b7af70 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:30.229733 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1673aaf0-332d-4b15-b41b-fe88f5b7af70 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:30.239516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1673aaf0-332d-4b15-b41b-fe88f5b7af70 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:30.244397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc24476-6d93-458e-abae-d32036877967 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:30.245026 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc24476-6d93-458e-abae-d32036877967 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:30.245304 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc24476-6d93-458e-abae-d32036877967 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:30.245573 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc24476-6d93-458e-abae-d32036877967 admin admin] PATH_INFO: `/v3/users/8c4ced79c3ff4842b30b472acb555cd5` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:30.246830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc24476-6d93-458e-abae-d32036877967 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8c4ced79c3ff4842b30b472acb555cd5)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:30.248782 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc24476-6d93-458e-abae-d32036877967 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:30.448552 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4fc24476-6d93-458e-abae-d32036877967 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8c4ced79c3ff4842b30b472acb555cd5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:30.449220 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4fc24476-6d93-458e-abae-d32036877967 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8c4ced79c3ff4842b30b472acb555cd5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:30.449832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4fc24476-6d93-458e-abae-d32036877967 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8c4ced79c3ff4842b30b472acb555cd5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:30.457762 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4fc24476-6d93-458e-abae-d32036877967 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8c4ced79c3ff4842b30b472acb555cd5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:30.539179 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4fc24476-6d93-458e-abae-d32036877967 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8c4ced79c3ff4842b30b472acb555cd5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:30.558684 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1291/2590] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:53:30 2022] DELETE /identity/v3/users/8c4ced79c3ff4842b30b472acb555cd5 => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:53:30.854363 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4413e24c-72da-49fb-b167-2c5552b49ba8 service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:30.864877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4413e24c-72da-49fb-b167-2c5552b49ba8 service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:30.912839 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:30.926162 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2294bf-7b29-4560-8ded-974ab34d24db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:30.926812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2294bf-7b29-4560-8ded-974ab34d24db admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:30.927090 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2294bf-7b29-4560-8ded-974ab34d24db admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:30.927357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2294bf-7b29-4560-8ded-974ab34d24db admin admin] PATH_INFO: `/v3/projects/86e6b0501cb94028be07bddd2182638b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:30.928445 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2294bf-7b29-4560-8ded-974ab34d24db admin admin] RBAC: Authorizing `identity:delete_project(project_id=86e6b0501cb94028be07bddd2182638b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:30.930377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2294bf-7b29-4560-8ded-974ab34d24db admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:31.041992 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0d2294bf-7b29-4560-8ded-974ab34d24db admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '86e6b0501cb94028be07bddd2182638b', 'request_id': 'req-0d2294bf-7b29-4560-8ded-974ab34d24db'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:31.047509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0d2294bf-7b29-4560-8ded-974ab34d24db admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '86e6b0501cb94028be07bddd2182638b', 'request_id': 'req-0d2294bf-7b29-4560-8ded-974ab34d24db'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:31.054296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0d2294bf-7b29-4560-8ded-974ab34d24db admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '86e6b0501cb94028be07bddd2182638b', 'request_id': 'req-0d2294bf-7b29-4560-8ded-974ab34d24db'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:31.103276 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0d2294bf-7b29-4560-8ded-974ab34d24db admin admin] The token cache is being invalidate because project 86e6b0501cb94028be07bddd2182638b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:53:31.103527 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0d2294bf-7b29-4560-8ded-974ab34d24db admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:31.104154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0d2294bf-7b29-4560-8ded-974ab34d24db admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:31.105667 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1300/2591] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:53:30 2022] DELETE /identity/v3/projects/86e6b0501cb94028be07bddd2182638b => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:31.110615 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3916f0-7051-4ee2-a598-fc3f496cbea3 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:31.110906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3916f0-7051-4ee2-a598-fc3f496cbea3 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:31.111179 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3916f0-7051-4ee2-a598-fc3f496cbea3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:31.149050 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-be3916f0-7051-4ee2-a598-fc3f496cbea3 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:31.150258 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-be3916f0-7051-4ee2-a598-fc3f496cbea3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:31.208541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1292/2592] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:53:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:53:31.216347 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdf54ba-32e5-4d75-b098-87c1d533113b None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:31.216633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdf54ba-32e5-4d75-b098-87c1d533113b None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:31.216931 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdf54ba-32e5-4d75-b098-87c1d533113b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:31.237323 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-8bdf54ba-32e5-4d75-b098-87c1d533113b None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:31.238519 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-8bdf54ba-32e5-4d75-b098-87c1d533113b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:31.246285 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1301/2593] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:53:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:53:31.250438 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3916f0-7051-4ee2-a598-fc3f496cbea3 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:31.260370 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3916f0-7051-4ee2-a598-fc3f496cbea3 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:31.264995 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d80a726-8a52-4cf2-8366-3fda836d175a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:31.265590 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d80a726-8a52-4cf2-8366-3fda836d175a admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:31.265870 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d80a726-8a52-4cf2-8366-3fda836d175a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:31.266146 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d80a726-8a52-4cf2-8366-3fda836d175a admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:31.266811 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d80a726-8a52-4cf2-8366-3fda836d175a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:53:31.267120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d80a726-8a52-4cf2-8366-3fda836d175a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:31.269245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d80a726-8a52-4cf2-8366-3fda836d175a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:31.281456 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1293/2594] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:53:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:53:31.286568 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdf54ba-32e5-4d75-b098-87c1d533113b None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:31.296031 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdf54ba-32e5-4d75-b098-87c1d533113b None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:31.300775 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2c3ac7-d76a-4c0a-9e3f-a8897e2499ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:31.301369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2c3ac7-d76a-4c0a-9e3f-a8897e2499ae admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:31.301648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2c3ac7-d76a-4c0a-9e3f-a8897e2499ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:31.301917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2c3ac7-d76a-4c0a-9e3f-a8897e2499ae admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:31.302534 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2c3ac7-d76a-4c0a-9e3f-a8897e2499ae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:31.304473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2c3ac7-d76a-4c0a-9e3f-a8897e2499ae admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:31.373958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1302/2595] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:53:31 2022] POST /identity/v3/projects => generated 405 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:53:31.378405 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d80a726-8a52-4cf2-8366-3fda836d175a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:31.388550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d80a726-8a52-4cf2-8366-3fda836d175a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:31.429836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:31.444144 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6c73ef-0b96-48c7-95f3-f460704a90ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:31.444746 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6c73ef-0b96-48c7-95f3-f460704a90ad admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:31.445065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6c73ef-0b96-48c7-95f3-f460704a90ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:31.445345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6c73ef-0b96-48c7-95f3-f460704a90ad admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:31.445980 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6c73ef-0b96-48c7-95f3-f460704a90ad admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:31.447933 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6c73ef-0b96-48c7-95f3-f460704a90ad admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:31.552516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1294/2596] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:53:31 2022] POST /identity/v3/users => generated 303 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:53:31.553839 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2c3ac7-d76a-4c0a-9e3f-a8897e2499ae admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:31.564622 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2c3ac7-d76a-4c0a-9e3f-a8897e2499ae admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:31.569485 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831cd24d-366b-4cd0-85a5-a5f47d13cdb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:31.570083 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831cd24d-366b-4cd0-85a5-a5f47d13cdb9 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:31.570364 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831cd24d-366b-4cd0-85a5-a5f47d13cdb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:31.570693 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831cd24d-366b-4cd0-85a5-a5f47d13cdb9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:31.571332 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831cd24d-366b-4cd0-85a5-a5f47d13cdb9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:31.573193 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831cd24d-366b-4cd0-85a5-a5f47d13cdb9 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:31.586986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1303/2597] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:53:31 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:53:31.588219 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6c73ef-0b96-48c7-95f3-f460704a90ad admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:31.598684 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6c73ef-0b96-48c7-95f3-f460704a90ad admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:31.603405 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2784ef-1265-4fa7-9435-1310cc5de08e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:31.604005 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2784ef-1265-4fa7-9435-1310cc5de08e admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:31.604287 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2784ef-1265-4fa7-9435-1310cc5de08e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:31.604561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2784ef-1265-4fa7-9435-1310cc5de08e admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:31.605192 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2784ef-1265-4fa7-9435-1310cc5de08e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:31.606838 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2784ef-1265-4fa7-9435-1310cc5de08e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:31.617516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1295/2598] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:53:31 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:53:31.621648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831cd24d-366b-4cd0-85a5-a5f47d13cdb9 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:31.632021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831cd24d-366b-4cd0-85a5-a5f47d13cdb9 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:31.636804 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d85d93-8fd4-4571-9a00-23eeadf23b0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:31.637379 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d85d93-8fd4-4571-9a00-23eeadf23b0c admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:31.637661 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d85d93-8fd4-4571-9a00-23eeadf23b0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:31.637938 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d85d93-8fd4-4571-9a00-23eeadf23b0c admin admin] PATH_INFO: `/v3/projects/dbda51b7423b4c4980a31fe5f1ecaef3/users/bb2df1d9c21f4b479e824a8e1c3603fd/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:31.652995 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d85d93-8fd4-4571-9a00-23eeadf23b0c admin admin] RBAC: Authorizing `identity:create_grant(project_id=dbda51b7423b4c4980a31fe5f1ecaef3, user_id=bb2df1d9c21f4b479e824a8e1c3603fd, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:31.655706 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d85d93-8fd4-4571-9a00-23eeadf23b0c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:31.721648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1304/2599] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:53:31 2022] PUT /identity/v3/projects/dbda51b7423b4c4980a31fe5f1ecaef3/users/bb2df1d9c21f4b479e824a8e1c3603fd/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:53:31.726701 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cc3929-ade8-408c-81ad-1d2dd2baf1f3 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:31.726985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cc3929-ade8-408c-81ad-1d2dd2baf1f3 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:31.727257 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cc3929-ade8-408c-81ad-1d2dd2baf1f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:31.746017 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f8cc3929-ade8-408c-81ad-1d2dd2baf1f3 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:31.747353 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f8cc3929-ade8-408c-81ad-1d2dd2baf1f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:31.805761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1296/2600] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:53:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:53:31.815875 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d85d93-8fd4-4571-9a00-23eeadf23b0c admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:31.817556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a3d85d93-8fd4-4571-9a00-23eeadf23b0c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:31.874421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d85d93-8fd4-4571-9a00-23eeadf23b0c admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:31.879584 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20e4eef-26f9-48de-b7bd-5f91f2e7e70f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:31.880123 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20e4eef-26f9-48de-b7bd-5f91f2e7e70f service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:31.880403 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20e4eef-26f9-48de-b7bd-5f91f2e7e70f service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:31.880675 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20e4eef-26f9-48de-b7bd-5f91f2e7e70f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:31.889485 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20e4eef-26f9-48de-b7bd-5f91f2e7e70f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:31.890954 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20e4eef-26f9-48de-b7bd-5f91f2e7e70f service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:31.900880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1305/2601] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:53:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:53:33.115536 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bce7606-a136-485f-9ce6-8969ddc4ee3c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:33.116047 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bce7606-a136-485f-9ce6-8969ddc4ee3c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:33.116166 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bce7606-a136-485f-9ce6-8969ddc4ee3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:33.146023 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-8bce7606-a136-485f-9ce6-8969ddc4ee3c None None] MFA Rules not processed for user `bb2df1d9c21f4b479e824a8e1c3603fd`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:33.147285 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8bce7606-a136-485f-9ce6-8969ddc4ee3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:33.223949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1297/2602] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:53:33 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:53:33.228636 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20e4eef-26f9-48de-b7bd-5f91f2e7e70f service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:33.238670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20e4eef-26f9-48de-b7bd-5f91f2e7e70f service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:33.243320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2676c16b-55c8-41f5-ba95-716108213551 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:33.243990 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2676c16b-55c8-41f5-ba95-716108213551 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:33.244366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2676c16b-55c8-41f5-ba95-716108213551 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:33.244642 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2676c16b-55c8-41f5-ba95-716108213551 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:33.245279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2676c16b-55c8-41f5-ba95-716108213551 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:33.247237 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2676c16b-55c8-41f5-ba95-716108213551 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:33.263056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:33.327082 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1306/2603] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:53:33 2022] POST /identity/v3/projects => generated 403 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:53:33.331917 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bce7606-a136-485f-9ce6-8969ddc4ee3c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:33.342738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bce7606-a136-485f-9ce6-8969ddc4ee3c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:33.389184 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:33.398439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137d4187-fb03-4358-af4d-f8bfd84dfa77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:33.399081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137d4187-fb03-4358-af4d-f8bfd84dfa77 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:33.399359 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137d4187-fb03-4358-af4d-f8bfd84dfa77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:33.399633 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137d4187-fb03-4358-af4d-f8bfd84dfa77 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:33.400255 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137d4187-fb03-4358-af4d-f8bfd84dfa77 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:33.402161 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137d4187-fb03-4358-af4d-f8bfd84dfa77 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:33.510210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1298/2604] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:53:33 2022] POST /identity/v3/users => generated 308 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:53:33.514141 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2676c16b-55c8-41f5-ba95-716108213551 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:33.525215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2676c16b-55c8-41f5-ba95-716108213551 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:33.533721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fff61d-a8c1-44d7-93c2-a59acd35bb9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:33.534327 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fff61d-a8c1-44d7-93c2-a59acd35bb9c admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:33.534641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fff61d-a8c1-44d7-93c2-a59acd35bb9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:33.534920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fff61d-a8c1-44d7-93c2-a59acd35bb9c admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:33.535558 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fff61d-a8c1-44d7-93c2-a59acd35bb9c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:33.537227 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fff61d-a8c1-44d7-93c2-a59acd35bb9c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:33.548016 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1307/2605] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:53:33 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:53:33.552628 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137d4187-fb03-4358-af4d-f8bfd84dfa77 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:33.562509 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137d4187-fb03-4358-af4d-f8bfd84dfa77 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:33.567856 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5530dc-5092-473c-bee6-dd020d25a658 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:33.568495 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5530dc-5092-473c-bee6-dd020d25a658 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:33.568797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5530dc-5092-473c-bee6-dd020d25a658 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:33.569029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5530dc-5092-473c-bee6-dd020d25a658 admin admin] PATH_INFO: `/v3/projects/5b550ae5406f4a87ac453fc2456c2c83/users/89bae3e30b2449a38ef78c27014c883e/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:33.583666 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5530dc-5092-473c-bee6-dd020d25a658 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5b550ae5406f4a87ac453fc2456c2c83, user_id=89bae3e30b2449a38ef78c27014c883e, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:33.586343 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5530dc-5092-473c-bee6-dd020d25a658 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:33.650544 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1299/2606] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:53:33 2022] PUT /identity/v3/projects/5b550ae5406f4a87ac453fc2456c2c83/users/89bae3e30b2449a38ef78c27014c883e/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:33.654459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fff61d-a8c1-44d7-93c2-a59acd35bb9c admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:33.664930 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fff61d-a8c1-44d7-93c2-a59acd35bb9c admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:33.720648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205ffaf5-fbd4-4adf-abc7-b2e35461f4b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:33.721251 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205ffaf5-fbd4-4adf-abc7-b2e35461f4b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:33.721534 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205ffaf5-fbd4-4adf-abc7-b2e35461f4b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:33.721807 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205ffaf5-fbd4-4adf-abc7-b2e35461f4b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:33.722434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-205ffaf5-fbd4-4adf-abc7-b2e35461f4b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:33.724069 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-205ffaf5-fbd4-4adf-abc7-b2e35461f4b6 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:33.733769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1308/2607] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:53:33 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:53:33.738408 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5530dc-5092-473c-bee6-dd020d25a658 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:33.748564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5530dc-5092-473c-bee6-dd020d25a658 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:33.753831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f05fdb-0099-4b30-9ec6-d2762d1e0819 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:33.754472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f05fdb-0099-4b30-9ec6-d2762d1e0819 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:33.754784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f05fdb-0099-4b30-9ec6-d2762d1e0819 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:33.755060 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f05fdb-0099-4b30-9ec6-d2762d1e0819 admin admin] PATH_INFO: `/v3/projects/5b550ae5406f4a87ac453fc2456c2c83/users/89bae3e30b2449a38ef78c27014c883e/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:33.756960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f05fdb-0099-4b30-9ec6-d2762d1e0819 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5b550ae5406f4a87ac453fc2456c2c83, user_id=89bae3e30b2449a38ef78c27014c883e, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:33.759642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f05fdb-0099-4b30-9ec6-d2762d1e0819 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:33.770236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1300/2608] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:53:33 2022] PUT /identity/v3/projects/5b550ae5406f4a87ac453fc2456c2c83/users/89bae3e30b2449a38ef78c27014c883e/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:34.973856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753d6c5e-3cca-46c6-b5ac-45c9553a8af9 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:34.974408 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753d6c5e-3cca-46c6-b5ac-45c9553a8af9 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:34.974510 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753d6c5e-3cca-46c6-b5ac-45c9553a8af9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:35.006710 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-753d6c5e-3cca-46c6-b5ac-45c9553a8af9 None None] MFA Rules not processed for user `89bae3e30b2449a38ef78c27014c883e`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:35.007969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-753d6c5e-3cca-46c6-b5ac-45c9553a8af9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:35.085816 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1309/2609] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:53:34 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:53:35.626000 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daef4e6c-87fa-47d9-8c6f-0c71a650171c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:35.626501 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daef4e6c-87fa-47d9-8c6f-0c71a650171c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:35.626630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daef4e6c-87fa-47d9-8c6f-0c71a650171c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:35.662827 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-daef4e6c-87fa-47d9-8c6f-0c71a650171c None None] MFA Rules not processed for user `7d6e0545a40a436d8c63820b795551a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:35.662827 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-daef4e6c-87fa-47d9-8c6f-0c71a650171c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:35.729655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1301/2610] 10.222.0.39 () {58 vars in 1009 bytes} [Fri Sep 30 09:53:35 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:53:35.908934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753d6c5e-3cca-46c6-b5ac-45c9553a8af9 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:35.910187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-753d6c5e-3cca-46c6-b5ac-45c9553a8af9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:35.980908 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753d6c5e-3cca-46c6-b5ac-45c9553a8af9 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:35.985936 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38be770b-175c-4b80-b49f-ef16079fb3e7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e5ad921e6fa41c297696ec7279f0fc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:35.987093 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38be770b-175c-4b80-b49f-ef16079fb3e7 service swift] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:35.987373 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38be770b-175c-4b80-b49f-ef16079fb3e7 service swift] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:35.987645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38be770b-175c-4b80-b49f-ef16079fb3e7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:35.992628 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:35.996394 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38be770b-175c-4b80-b49f-ef16079fb3e7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:35.998115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38be770b-175c-4b80-b49f-ef16079fb3e7 service swift] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:36.007367 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1310/2611] 10.222.0.39 () {60 vars in 1384 bytes} [Fri Sep 30 09:53:35 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 30 09:53:36.576105 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daef4e6c-87fa-47d9-8c6f-0c71a650171c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:36.658216 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daef4e6c-87fa-47d9-8c6f-0c71a650171c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:36.663199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18b9072-660e-40b8-a18b-5c0c330e9636 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:36.664047 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18b9072-660e-40b8-a18b-5c0c330e9636 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:36.664487 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18b9072-660e-40b8-a18b-5c0c330e9636 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:36.664931 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18b9072-660e-40b8-a18b-5c0c330e9636 admin admin] PATH_INFO: `/v3/users/bb2df1d9c21f4b479e824a8e1c3603fd` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:36.666525 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18b9072-660e-40b8-a18b-5c0c330e9636 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bb2df1d9c21f4b479e824a8e1c3603fd)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:36.668675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18b9072-660e-40b8-a18b-5c0c330e9636 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:36.866798 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:INSERT=5,keystone:DELETE=6 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:36.869048 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d18b9072-660e-40b8-a18b-5c0c330e9636 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bb2df1d9c21f4b479e824a8e1c3603fd'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:36.869711 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d18b9072-660e-40b8-a18b-5c0c330e9636 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bb2df1d9c21f4b479e824a8e1c3603fd'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:36.870334 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d18b9072-660e-40b8-a18b-5c0c330e9636 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bb2df1d9c21f4b479e824a8e1c3603fd'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:36.878384 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d18b9072-660e-40b8-a18b-5c0c330e9636 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bb2df1d9c21f4b479e824a8e1c3603fd'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:36.976987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d18b9072-660e-40b8-a18b-5c0c330e9636 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bb2df1d9c21f4b479e824a8e1c3603fd'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:36.998394 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1302/2612] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:53:36 2022] DELETE /identity/v3/users/bb2df1d9c21f4b479e824a8e1c3603fd => generated 0 bytes in 423 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:37.277603 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38be770b-175c-4b80-b49f-ef16079fb3e7 service swift] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:37.290298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38be770b-175c-4b80-b49f-ef16079fb3e7 service swift] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:37.385221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8452bb44-51bb-4c68-9951-da2fc3754f30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:37.386446 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8452bb44-51bb-4c68-9951-da2fc3754f30 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:37.386955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8452bb44-51bb-4c68-9951-da2fc3754f30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:37.387500 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8452bb44-51bb-4c68-9951-da2fc3754f30 admin admin] PATH_INFO: `/v3/projects/dbda51b7423b4c4980a31fe5f1ecaef3` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:37.391362 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8452bb44-51bb-4c68-9951-da2fc3754f30 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dbda51b7423b4c4980a31fe5f1ecaef3)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:37.394147 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8452bb44-51bb-4c68-9951-da2fc3754f30 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:37.504056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8452bb44-51bb-4c68-9951-da2fc3754f30 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dbda51b7423b4c4980a31fe5f1ecaef3', 'request_id': 'req-8452bb44-51bb-4c68-9951-da2fc3754f30'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:37.513499 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8452bb44-51bb-4c68-9951-da2fc3754f30 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dbda51b7423b4c4980a31fe5f1ecaef3', 'request_id': 'req-8452bb44-51bb-4c68-9951-da2fc3754f30'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:37.520577 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8452bb44-51bb-4c68-9951-da2fc3754f30 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dbda51b7423b4c4980a31fe5f1ecaef3', 'request_id': 'req-8452bb44-51bb-4c68-9951-da2fc3754f30'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:37.610858 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8452bb44-51bb-4c68-9951-da2fc3754f30 admin admin] The token cache is being invalidate because project dbda51b7423b4c4980a31fe5f1ecaef3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:53:37.611380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8452bb44-51bb-4c68-9951-da2fc3754f30 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:37.616862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8452bb44-51bb-4c68-9951-da2fc3754f30 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:37.616862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1311/2613] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:53:37 2022] DELETE /identity/v3/projects/dbda51b7423b4c4980a31fe5f1ecaef3 => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:37.620156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18b9072-660e-40b8-a18b-5c0c330e9636 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:37.621873 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d18b9072-660e-40b8-a18b-5c0c330e9636 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:37.646489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18b9072-660e-40b8-a18b-5c0c330e9636 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:37.711688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f91104c-5267-4821-b007-e98f62435bf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:37.713164 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f91104c-5267-4821-b007-e98f62435bf8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:37.713615 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f91104c-5267-4821-b007-e98f62435bf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:37.713893 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f91104c-5267-4821-b007-e98f62435bf8 admin admin] PATH_INFO: `/v3/users/89bae3e30b2449a38ef78c27014c883e` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:37.715110 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f91104c-5267-4821-b007-e98f62435bf8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=89bae3e30b2449a38ef78c27014c883e)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:37.717212 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f91104c-5267-4821-b007-e98f62435bf8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:37.940761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4f91104c-5267-4821-b007-e98f62435bf8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89bae3e30b2449a38ef78c27014c883e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:37.941568 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4f91104c-5267-4821-b007-e98f62435bf8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89bae3e30b2449a38ef78c27014c883e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:37.942231 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4f91104c-5267-4821-b007-e98f62435bf8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89bae3e30b2449a38ef78c27014c883e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:37.949918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4f91104c-5267-4821-b007-e98f62435bf8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89bae3e30b2449a38ef78c27014c883e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:38.017934 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4f91104c-5267-4821-b007-e98f62435bf8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89bae3e30b2449a38ef78c27014c883e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:38.042766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1303/2614] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:53:37 2022] DELETE /identity/v3/users/89bae3e30b2449a38ef78c27014c883e => generated 0 bytes in 423 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:38.312614 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8452bb44-51bb-4c68-9951-da2fc3754f30 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:38.312614 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-8452bb44-51bb-4c68-9951-da2fc3754f30 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:38.327343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8452bb44-51bb-4c68-9951-da2fc3754f30 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:38.401233 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f9222d-5c88-4e88-97f0-b0cf5122e659 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:38.401963 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f9222d-5c88-4e88-97f0-b0cf5122e659 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:38.402248 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f9222d-5c88-4e88-97f0-b0cf5122e659 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:38.402524 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f9222d-5c88-4e88-97f0-b0cf5122e659 admin admin] PATH_INFO: `/v3/projects/5b550ae5406f4a87ac453fc2456c2c83` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:38.404286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f9222d-5c88-4e88-97f0-b0cf5122e659 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5b550ae5406f4a87ac453fc2456c2c83)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:38.407661 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f9222d-5c88-4e88-97f0-b0cf5122e659 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:38.627544 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:38.648325 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a6f9222d-5c88-4e88-97f0-b0cf5122e659 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5b550ae5406f4a87ac453fc2456c2c83', 'request_id': 'req-a6f9222d-5c88-4e88-97f0-b0cf5122e659'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:38.654753 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a6f9222d-5c88-4e88-97f0-b0cf5122e659 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5b550ae5406f4a87ac453fc2456c2c83', 'request_id': 'req-a6f9222d-5c88-4e88-97f0-b0cf5122e659'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:38.661852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a6f9222d-5c88-4e88-97f0-b0cf5122e659 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5b550ae5406f4a87ac453fc2456c2c83', 'request_id': 'req-a6f9222d-5c88-4e88-97f0-b0cf5122e659'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:38.744615 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a6f9222d-5c88-4e88-97f0-b0cf5122e659 admin admin] The token cache is being invalidate because project 5b550ae5406f4a87ac453fc2456c2c83 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:53:38.744955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a6f9222d-5c88-4e88-97f0-b0cf5122e659 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:38.745591 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-a6f9222d-5c88-4e88-97f0-b0cf5122e659 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:38.750668 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1312/2615] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:53:38 2022] DELETE /identity/v3/projects/5b550ae5406f4a87ac453fc2456c2c83 => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:38.752987 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ee73c7-7128-4569-84eb-236198b74473 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:38.753455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ee73c7-7128-4569-84eb-236198b74473 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:38.753550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ee73c7-7128-4569-84eb-236198b74473 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:38.779363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-16ee73c7-7128-4569-84eb-236198b74473 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:38.779363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-16ee73c7-7128-4569-84eb-236198b74473 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:38.815510 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:38.842000 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1304/2616] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:53:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:53:38.849815 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e047e6-eaf8-48b2-bdf7-4db556b172cb None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:38.850098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e047e6-eaf8-48b2-bdf7-4db556b172cb None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:38.850469 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e047e6-eaf8-48b2-bdf7-4db556b172cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:38.872175 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-d0e047e6-eaf8-48b2-bdf7-4db556b172cb None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:38.874009 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d0e047e6-eaf8-48b2-bdf7-4db556b172cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:38.886958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1313/2617] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:53:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:53:38.888256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ee73c7-7128-4569-84eb-236198b74473 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:38.898215 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ee73c7-7128-4569-84eb-236198b74473 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:38.903120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02eeb96-b84e-4a28-951c-2be8385f2a27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:38.903776 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02eeb96-b84e-4a28-951c-2be8385f2a27 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:38.904090 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02eeb96-b84e-4a28-951c-2be8385f2a27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:38.904335 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02eeb96-b84e-4a28-951c-2be8385f2a27 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:38.905013 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02eeb96-b84e-4a28-951c-2be8385f2a27 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:53:38.905332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02eeb96-b84e-4a28-951c-2be8385f2a27 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:38.907084 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02eeb96-b84e-4a28-951c-2be8385f2a27 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:38.919720 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1305/2618] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:53:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:53:38.924753 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e047e6-eaf8-48b2-bdf7-4db556b172cb None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:38.934917 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e047e6-eaf8-48b2-bdf7-4db556b172cb None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:38.940412 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3006eb74-f073-4793-94c3-ad0ac563da9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:38.941239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3006eb74-f073-4793-94c3-ad0ac563da9c admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:38.941352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3006eb74-f073-4793-94c3-ad0ac563da9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:38.941615 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3006eb74-f073-4793-94c3-ad0ac563da9c admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:38.942257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3006eb74-f073-4793-94c3-ad0ac563da9c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:38.944334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3006eb74-f073-4793-94c3-ad0ac563da9c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:39.275863 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1314/2619] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:53:38 2022] POST /identity/v3/projects => generated 391 bytes in 352 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:53:39.280034 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02eeb96-b84e-4a28-951c-2be8385f2a27 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:39.290506 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02eeb96-b84e-4a28-951c-2be8385f2a27 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:39.346426 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1196f186-8d77-47da-99f4-4f72eecc87ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:39.347121 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1196f186-8d77-47da-99f4-4f72eecc87ac admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:39.347403 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1196f186-8d77-47da-99f4-4f72eecc87ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:39.347672 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1196f186-8d77-47da-99f4-4f72eecc87ac admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:39.348324 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1196f186-8d77-47da-99f4-4f72eecc87ac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:39.350299 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1196f186-8d77-47da-99f4-4f72eecc87ac admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:39.436051 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1306/2620] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:53:39 2022] POST /identity/v3/users => generated 296 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:53:39.440488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3006eb74-f073-4793-94c3-ad0ac563da9c admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:39.453061 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3006eb74-f073-4793-94c3-ad0ac563da9c admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:39.456308 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f98f0dd-b65a-46bc-a91f-5d0c6f237dca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:39.457056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f98f0dd-b65a-46bc-a91f-5d0c6f237dca admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:39.457428 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f98f0dd-b65a-46bc-a91f-5d0c6f237dca admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:39.457797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f98f0dd-b65a-46bc-a91f-5d0c6f237dca admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:39.458528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f98f0dd-b65a-46bc-a91f-5d0c6f237dca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:39.460275 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f98f0dd-b65a-46bc-a91f-5d0c6f237dca admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:39.473092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1315/2621] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:53:39 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:53:39.477419 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1196f186-8d77-47da-99f4-4f72eecc87ac admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:39.490939 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1196f186-8d77-47da-99f4-4f72eecc87ac admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:39.495838 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a8b3cd-9f86-4bfd-908f-45fe328461f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:39.496438 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a8b3cd-9f86-4bfd-908f-45fe328461f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:39.496716 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a8b3cd-9f86-4bfd-908f-45fe328461f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:39.497035 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a8b3cd-9f86-4bfd-908f-45fe328461f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:39.497698 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a8b3cd-9f86-4bfd-908f-45fe328461f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:39.500707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a8b3cd-9f86-4bfd-908f-45fe328461f6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:39.514649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1307/2622] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:53:39 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:53:39.515801 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f98f0dd-b65a-46bc-a91f-5d0c6f237dca admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:39.527174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f98f0dd-b65a-46bc-a91f-5d0c6f237dca admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:39.532407 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479fb0db-b88a-4fe3-8fea-b43c5684a68a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:39.533816 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479fb0db-b88a-4fe3-8fea-b43c5684a68a admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:39.534105 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479fb0db-b88a-4fe3-8fea-b43c5684a68a admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:39.534404 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479fb0db-b88a-4fe3-8fea-b43c5684a68a admin admin] PATH_INFO: `/v3/projects/cd57b0eceece4299b37d3aef570cb197/users/a4b61509680c4d15b17bad0408cb7dd4/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:39.550040 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479fb0db-b88a-4fe3-8fea-b43c5684a68a admin admin] RBAC: Authorizing `identity:create_grant(project_id=cd57b0eceece4299b37d3aef570cb197, user_id=a4b61509680c4d15b17bad0408cb7dd4, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:39.552754 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479fb0db-b88a-4fe3-8fea-b43c5684a68a admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:39.623361 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1316/2623] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:53:39 2022] PUT /identity/v3/projects/cd57b0eceece4299b37d3aef570cb197/users/a4b61509680c4d15b17bad0408cb7dd4/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:53:39.636516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41513700-443f-4eab-97ae-7382f877b6ec None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:39.636516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41513700-443f-4eab-97ae-7382f877b6ec None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:39.636516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41513700-443f-4eab-97ae-7382f877b6ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:39.669558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-41513700-443f-4eab-97ae-7382f877b6ec None None] MFA Rules not processed for user `a4b61509680c4d15b17bad0408cb7dd4`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:39.671114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-41513700-443f-4eab-97ae-7382f877b6ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:39.747475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1308/2624] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:53:39 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:53:39.751880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479fb0db-b88a-4fe3-8fea-b43c5684a68a admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:39.821694 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479fb0db-b88a-4fe3-8fea-b43c5684a68a admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:39.828067 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba234e8-bcc4-4848-a5e0-b1ab5cd94eae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:39.828697 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba234e8-bcc4-4848-a5e0-b1ab5cd94eae admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:39.828980 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba234e8-bcc4-4848-a5e0-b1ab5cd94eae admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:39.829284 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba234e8-bcc4-4848-a5e0-b1ab5cd94eae admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:39.830473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba234e8-bcc4-4848-a5e0-b1ab5cd94eae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:39.832455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba234e8-bcc4-4848-a5e0-b1ab5cd94eae admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:39.930649 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1317/2625] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:53:39 2022] POST /identity/v3/projects => generated 391 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:53:39.932997 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41513700-443f-4eab-97ae-7382f877b6ec None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:39.948650 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41513700-443f-4eab-97ae-7382f877b6ec None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:39.990813 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:40.016312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8697e2-78db-4340-94be-7089912a49d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:40.017022 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8697e2-78db-4340-94be-7089912a49d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:40.017336 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8697e2-78db-4340-94be-7089912a49d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:40.017641 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8697e2-78db-4340-94be-7089912a49d6 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:40.018272 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8697e2-78db-4340-94be-7089912a49d6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:40.020289 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8697e2-78db-4340-94be-7089912a49d6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:40.110624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1309/2626] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:53:39 2022] POST /identity/v3/users => generated 302 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:53:40.113360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba234e8-bcc4-4848-a5e0-b1ab5cd94eae admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:40.124612 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba234e8-bcc4-4848-a5e0-b1ab5cd94eae admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:40.131415 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746f95e2-e711-4e67-bb3d-1398cf2af5d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:40.132213 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746f95e2-e711-4e67-bb3d-1398cf2af5d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:40.132594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746f95e2-e711-4e67-bb3d-1398cf2af5d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:40.132966 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746f95e2-e711-4e67-bb3d-1398cf2af5d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:40.133715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746f95e2-e711-4e67-bb3d-1398cf2af5d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:40.135579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746f95e2-e711-4e67-bb3d-1398cf2af5d3 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:40.147653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1318/2627] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:53:40 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:53:40.153642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8697e2-78db-4340-94be-7089912a49d6 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:40.167248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8697e2-78db-4340-94be-7089912a49d6 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:40.173015 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879c4349-123a-4edb-997d-63185c35ba80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:40.182967 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879c4349-123a-4edb-997d-63185c35ba80 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:40.182967 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879c4349-123a-4edb-997d-63185c35ba80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:40.182967 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879c4349-123a-4edb-997d-63185c35ba80 admin admin] PATH_INFO: `/v3/projects/b08fcad561da4e5e9c6de182db49f9cc/users/0e54bbfb4287489ba8d374389fd30f9b/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:40.213647 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879c4349-123a-4edb-997d-63185c35ba80 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b08fcad561da4e5e9c6de182db49f9cc, user_id=0e54bbfb4287489ba8d374389fd30f9b, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:40.216917 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879c4349-123a-4edb-997d-63185c35ba80 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:40.315946 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1310/2628] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:53:40 2022] PUT /identity/v3/projects/b08fcad561da4e5e9c6de182db49f9cc/users/0e54bbfb4287489ba8d374389fd30f9b/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:40.319563 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746f95e2-e711-4e67-bb3d-1398cf2af5d3 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:40.330809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746f95e2-e711-4e67-bb3d-1398cf2af5d3 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:40.362394 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:40.413063 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad33017f-08fc-467d-b60f-97ff52a3e893 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:40.413687 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad33017f-08fc-467d-b60f-97ff52a3e893 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:40.413966 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad33017f-08fc-467d-b60f-97ff52a3e893 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:40.414239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad33017f-08fc-467d-b60f-97ff52a3e893 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:40.414924 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad33017f-08fc-467d-b60f-97ff52a3e893 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:40.416686 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad33017f-08fc-467d-b60f-97ff52a3e893 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:40.432145 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1319/2629] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:53:40 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:53:40.436492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879c4349-123a-4edb-997d-63185c35ba80 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:40.448451 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879c4349-123a-4edb-997d-63185c35ba80 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:40.454144 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2450bc0a-1d12-419b-8348-c60837f0bd53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:40.454872 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2450bc0a-1d12-419b-8348-c60837f0bd53 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:40.455154 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2450bc0a-1d12-419b-8348-c60837f0bd53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:40.455432 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2450bc0a-1d12-419b-8348-c60837f0bd53 admin admin] PATH_INFO: `/v3/projects/b08fcad561da4e5e9c6de182db49f9cc/users/0e54bbfb4287489ba8d374389fd30f9b/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:40.457863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2450bc0a-1d12-419b-8348-c60837f0bd53 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b08fcad561da4e5e9c6de182db49f9cc, user_id=0e54bbfb4287489ba8d374389fd30f9b, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:40.460755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2450bc0a-1d12-419b-8348-c60837f0bd53 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:40.477537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1311/2630] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:53:40 2022] PUT /identity/v3/projects/b08fcad561da4e5e9c6de182db49f9cc/users/0e54bbfb4287489ba8d374389fd30f9b/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:40.488128 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5642049-b078-4ba0-92f1-c071afee8019 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:40.488675 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5642049-b078-4ba0-92f1-c071afee8019 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:40.488786 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5642049-b078-4ba0-92f1-c071afee8019 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:40.520845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-d5642049-b078-4ba0-92f1-c071afee8019 None None] MFA Rules not processed for user `0e54bbfb4287489ba8d374389fd30f9b`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:40.522064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d5642049-b078-4ba0-92f1-c071afee8019 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:40.598362 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1320/2631] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:53:40 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:53:40.610021 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2450bc0a-1d12-419b-8348-c60837f0bd53 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:40.612712 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2450bc0a-1d12-419b-8348-c60837f0bd53 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:40.722152 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2450bc0a-1d12-419b-8348-c60837f0bd53 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:40.726114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51551a25-7e1b-48f3-8459-7c60d14be31a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:40.726833 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51551a25-7e1b-48f3-8459-7c60d14be31a service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:40.727182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51551a25-7e1b-48f3-8459-7c60d14be31a service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:40.727524 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51551a25-7e1b-48f3-8459-7c60d14be31a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:40.742063 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51551a25-7e1b-48f3-8459-7c60d14be31a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:40.743756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51551a25-7e1b-48f3-8459-7c60d14be31a service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:40.758779 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1312/2632] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:53:40 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:53:40.817541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5642049-b078-4ba0-92f1-c071afee8019 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:40.819195 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d5642049-b078-4ba0-92f1-c071afee8019 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:40.832662 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5642049-b078-4ba0-92f1-c071afee8019 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:40.838376 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3614c0-c813-432c-a604-d9a8ef667e80 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:40.838376 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3614c0-c813-432c-a604-d9a8ef667e80 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:40.838376 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3614c0-c813-432c-a604-d9a8ef667e80 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:40.838376 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3614c0-c813-432c-a604-d9a8ef667e80 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:40.907270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3614c0-c813-432c-a604-d9a8ef667e80 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:40.908682 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3614c0-c813-432c-a604-d9a8ef667e80 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:40.918819 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1321/2633] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:53:40 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:53:46.074738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51551a25-7e1b-48f3-8459-7c60d14be31a service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:46.157270 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51551a25-7e1b-48f3-8459-7c60d14be31a service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:46.165072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d450628-fd86-4f26-8363-0632f979fef7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:46.166293 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d450628-fd86-4f26-8363-0632f979fef7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:46.166581 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d450628-fd86-4f26-8363-0632f979fef7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:46.166880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d450628-fd86-4f26-8363-0632f979fef7 admin admin] PATH_INFO: `/v3/users/a4b61509680c4d15b17bad0408cb7dd4` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:46.172116 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d450628-fd86-4f26-8363-0632f979fef7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a4b61509680c4d15b17bad0408cb7dd4)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:46.178582 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d450628-fd86-4f26-8363-0632f979fef7 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:46.203243 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:46.387254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0d450628-fd86-4f26-8363-0632f979fef7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a4b61509680c4d15b17bad0408cb7dd4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:46.388220 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0d450628-fd86-4f26-8363-0632f979fef7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a4b61509680c4d15b17bad0408cb7dd4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:46.388963 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0d450628-fd86-4f26-8363-0632f979fef7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a4b61509680c4d15b17bad0408cb7dd4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:46.397271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0d450628-fd86-4f26-8363-0632f979fef7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a4b61509680c4d15b17bad0408cb7dd4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:46.459954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0d450628-fd86-4f26-8363-0632f979fef7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a4b61509680c4d15b17bad0408cb7dd4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:46.480231 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1313/2634] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:53:46 2022] DELETE /identity/v3/users/a4b61509680c4d15b17bad0408cb7dd4 => generated 0 bytes in 406 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:46.484722 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dff67a9-4f3b-42e8-a681-9a2bbcf080ce None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:46.485528 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dff67a9-4f3b-42e8-a681-9a2bbcf080ce None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:46.486009 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dff67a9-4f3b-42e8-a681-9a2bbcf080ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:46.508735 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-9dff67a9-4f3b-42e8-a681-9a2bbcf080ce None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:46.510248 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-9dff67a9-4f3b-42e8-a681-9a2bbcf080ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:46.555833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:46.578778 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1322/2635] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:53:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:53:46.586320 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d450628-fd86-4f26-8363-0632f979fef7 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:46.587877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0d450628-fd86-4f26-8363-0632f979fef7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:46.662453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d450628-fd86-4f26-8363-0632f979fef7 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:46.666248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16127c19-76b8-4e85-9aa8-75c498d5d184 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:46.666913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16127c19-76b8-4e85-9aa8-75c498d5d184 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:46.667220 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16127c19-76b8-4e85-9aa8-75c498d5d184 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:46.667497 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16127c19-76b8-4e85-9aa8-75c498d5d184 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:46.677179 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16127c19-76b8-4e85-9aa8-75c498d5d184 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:46.678633 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16127c19-76b8-4e85-9aa8-75c498d5d184 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:46.687964 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1314/2636] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:53:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:53:47.111814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dff67a9-4f3b-42e8-a681-9a2bbcf080ce None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:47.113355 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-9dff67a9-4f3b-42e8-a681-9a2bbcf080ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:47.126903 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dff67a9-4f3b-42e8-a681-9a2bbcf080ce None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:47.132054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc25050-e20f-4b2f-93b7-850ef565c20f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:47.132692 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc25050-e20f-4b2f-93b7-850ef565c20f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:47.132977 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc25050-e20f-4b2f-93b7-850ef565c20f admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:47.133254 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc25050-e20f-4b2f-93b7-850ef565c20f admin admin] PATH_INFO: `/v3/projects/cd57b0eceece4299b37d3aef570cb197` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:47.134372 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc25050-e20f-4b2f-93b7-850ef565c20f admin admin] RBAC: Authorizing `identity:delete_project(project_id=cd57b0eceece4299b37d3aef570cb197)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:47.136469 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc25050-e20f-4b2f-93b7-850ef565c20f admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:47.282879 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-2bc25050-e20f-4b2f-93b7-850ef565c20f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cd57b0eceece4299b37d3aef570cb197', 'request_id': 'req-2bc25050-e20f-4b2f-93b7-850ef565c20f'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:47.290453 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-2bc25050-e20f-4b2f-93b7-850ef565c20f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cd57b0eceece4299b37d3aef570cb197', 'request_id': 'req-2bc25050-e20f-4b2f-93b7-850ef565c20f'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:47.296635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-2bc25050-e20f-4b2f-93b7-850ef565c20f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cd57b0eceece4299b37d3aef570cb197', 'request_id': 'req-2bc25050-e20f-4b2f-93b7-850ef565c20f'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:47.367366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-2bc25050-e20f-4b2f-93b7-850ef565c20f admin admin] The token cache is being invalidate because project cd57b0eceece4299b37d3aef570cb197 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:53:47.367658 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-2bc25050-e20f-4b2f-93b7-850ef565c20f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:47.368319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-2bc25050-e20f-4b2f-93b7-850ef565c20f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:47.369819 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1323/2637] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:53:47 2022] DELETE /identity/v3/projects/cd57b0eceece4299b37d3aef570cb197 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:47.373797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16127c19-76b8-4e85-9aa8-75c498d5d184 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:47.384441 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16127c19-76b8-4e85-9aa8-75c498d5d184 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:47.442704 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8c2362-e21b-438b-ae27-84636e845458 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:47.443333 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8c2362-e21b-438b-ae27-84636e845458 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:47.443615 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8c2362-e21b-438b-ae27-84636e845458 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:47.443890 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8c2362-e21b-438b-ae27-84636e845458 admin admin] PATH_INFO: `/v3/users/0e54bbfb4287489ba8d374389fd30f9b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:47.445025 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8c2362-e21b-438b-ae27-84636e845458 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e54bbfb4287489ba8d374389fd30f9b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:47.446986 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8c2362-e21b-438b-ae27-84636e845458 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:47.618535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:47.629109 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1c8c2362-e21b-438b-ae27-84636e845458 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e54bbfb4287489ba8d374389fd30f9b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:47.629764 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1c8c2362-e21b-438b-ae27-84636e845458 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e54bbfb4287489ba8d374389fd30f9b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:47.630365 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1c8c2362-e21b-438b-ae27-84636e845458 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e54bbfb4287489ba8d374389fd30f9b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:47.637212 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1c8c2362-e21b-438b-ae27-84636e845458 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e54bbfb4287489ba8d374389fd30f9b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:47.721195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1c8c2362-e21b-438b-ae27-84636e845458 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e54bbfb4287489ba8d374389fd30f9b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:47.739171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1315/2638] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:53:47 2022] DELETE /identity/v3/users/0e54bbfb4287489ba8d374389fd30f9b => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:48.155750 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc25050-e20f-4b2f-93b7-850ef565c20f admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:48.157796 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-2bc25050-e20f-4b2f-93b7-850ef565c20f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:48.174625 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc25050-e20f-4b2f-93b7-850ef565c20f admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:48.241073 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f489bd9-6385-4e32-9d4e-b4bd8f25d3e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:48.241709 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f489bd9-6385-4e32-9d4e-b4bd8f25d3e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:48.241996 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f489bd9-6385-4e32-9d4e-b4bd8f25d3e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:48.242270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f489bd9-6385-4e32-9d4e-b4bd8f25d3e4 admin admin] PATH_INFO: `/v3/projects/b08fcad561da4e5e9c6de182db49f9cc` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:48.243458 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f489bd9-6385-4e32-9d4e-b4bd8f25d3e4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b08fcad561da4e5e9c6de182db49f9cc)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:48.245371 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f489bd9-6385-4e32-9d4e-b4bd8f25d3e4 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:48.351504 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-1f489bd9-6385-4e32-9d4e-b4bd8f25d3e4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b08fcad561da4e5e9c6de182db49f9cc', 'request_id': 'req-1f489bd9-6385-4e32-9d4e-b4bd8f25d3e4'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:48.356887 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-1f489bd9-6385-4e32-9d4e-b4bd8f25d3e4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b08fcad561da4e5e9c6de182db49f9cc', 'request_id': 'req-1f489bd9-6385-4e32-9d4e-b4bd8f25d3e4'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:48.363680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-1f489bd9-6385-4e32-9d4e-b4bd8f25d3e4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b08fcad561da4e5e9c6de182db49f9cc', 'request_id': 'req-1f489bd9-6385-4e32-9d4e-b4bd8f25d3e4'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:48.420033 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:48.425746 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-1f489bd9-6385-4e32-9d4e-b4bd8f25d3e4 admin admin] The token cache is being invalidate because project b08fcad561da4e5e9c6de182db49f9cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:53:48.426108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-1f489bd9-6385-4e32-9d4e-b4bd8f25d3e4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:48.426878 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-1f489bd9-6385-4e32-9d4e-b4bd8f25d3e4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:48.431457 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1324/2639] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:53:48 2022] DELETE /identity/v3/projects/b08fcad561da4e5e9c6de182db49f9cc => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:48.433974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f9b5ed-4045-47d6-9851-be86f34bdc1d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:48.434398 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f9b5ed-4045-47d6-9851-be86f34bdc1d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:48.434563 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f9b5ed-4045-47d6-9851-be86f34bdc1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:48.454945 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d6f9b5ed-4045-47d6-9851-be86f34bdc1d None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:48.458046 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d6f9b5ed-4045-47d6-9851-be86f34bdc1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:48.518318 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1316/2640] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:53:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:53:48.525863 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3f25f9-d482-496e-8a8f-fdfdafa7c3c8 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:48.526121 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3f25f9-d482-496e-8a8f-fdfdafa7c3c8 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:48.526471 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3f25f9-d482-496e-8a8f-fdfdafa7c3c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:48.548867 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-ad3f25f9-d482-496e-8a8f-fdfdafa7c3c8 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:48.550101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ad3f25f9-d482-496e-8a8f-fdfdafa7c3c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:48.557952 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1325/2641] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:53:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:53:48.562195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f9b5ed-4045-47d6-9851-be86f34bdc1d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:48.572151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f9b5ed-4045-47d6-9851-be86f34bdc1d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:48.576959 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28268402-b152-48cc-9708-a144c929cad4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:48.577612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28268402-b152-48cc-9708-a144c929cad4 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:48.577964 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28268402-b152-48cc-9708-a144c929cad4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:48.578387 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28268402-b152-48cc-9708-a144c929cad4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:48.579099 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28268402-b152-48cc-9708-a144c929cad4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:53:48.579542 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28268402-b152-48cc-9708-a144c929cad4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:48.581673 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28268402-b152-48cc-9708-a144c929cad4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:48.593731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1317/2642] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:53:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:53:48.598520 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3f25f9-d482-496e-8a8f-fdfdafa7c3c8 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:48.607749 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3f25f9-d482-496e-8a8f-fdfdafa7c3c8 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:48.612879 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49f1494-d4e1-46f3-991a-26aa42403192 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:48.613462 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49f1494-d4e1-46f3-991a-26aa42403192 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:48.613740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49f1494-d4e1-46f3-991a-26aa42403192 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:48.614010 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49f1494-d4e1-46f3-991a-26aa42403192 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:48.614665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49f1494-d4e1-46f3-991a-26aa42403192 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:48.616588 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49f1494-d4e1-46f3-991a-26aa42403192 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:48.728664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1326/2643] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:53:48 2022] POST /identity/v3/projects => generated 379 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:53:48.732839 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28268402-b152-48cc-9708-a144c929cad4 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:48.743398 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28268402-b152-48cc-9708-a144c929cad4 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:48.798169 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80084dd9-c87c-42c3-9d2a-2aa031670554 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:48.798803 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80084dd9-c87c-42c3-9d2a-2aa031670554 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:48.799613 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80084dd9-c87c-42c3-9d2a-2aa031670554 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:48.799887 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80084dd9-c87c-42c3-9d2a-2aa031670554 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:48.800521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80084dd9-c87c-42c3-9d2a-2aa031670554 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:48.802417 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80084dd9-c87c-42c3-9d2a-2aa031670554 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:48.921785 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1318/2644] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:53:48 2022] POST /identity/v3/users => generated 290 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:53:48.926426 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49f1494-d4e1-46f3-991a-26aa42403192 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:48.936451 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49f1494-d4e1-46f3-991a-26aa42403192 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:48.940983 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b043aba-5a9a-4939-93ce-e3e4f6005f43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:48.941564 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b043aba-5a9a-4939-93ce-e3e4f6005f43 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:48.941840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b043aba-5a9a-4939-93ce-e3e4f6005f43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:48.942106 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b043aba-5a9a-4939-93ce-e3e4f6005f43 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:48.942767 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b043aba-5a9a-4939-93ce-e3e4f6005f43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:48.944350 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b043aba-5a9a-4939-93ce-e3e4f6005f43 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:48.954021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1327/2645] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:53:48 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:53:48.957931 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80084dd9-c87c-42c3-9d2a-2aa031670554 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:48.967851 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80084dd9-c87c-42c3-9d2a-2aa031670554 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:48.973137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca06061-2260-4367-bbf2-e35ae1cd8598 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:48.973755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca06061-2260-4367-bbf2-e35ae1cd8598 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:48.974031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca06061-2260-4367-bbf2-e35ae1cd8598 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:48.974302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca06061-2260-4367-bbf2-e35ae1cd8598 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:48.974997 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca06061-2260-4367-bbf2-e35ae1cd8598 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:48.976713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca06061-2260-4367-bbf2-e35ae1cd8598 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:48.989023 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1319/2646] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:53:48 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:53:48.991905 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b043aba-5a9a-4939-93ce-e3e4f6005f43 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:49.001744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b043aba-5a9a-4939-93ce-e3e4f6005f43 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:49.007283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d06373-10f2-42ef-8df3-a7ee3e8f5327 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:49.007918 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d06373-10f2-42ef-8df3-a7ee3e8f5327 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:49.008197 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d06373-10f2-42ef-8df3-a7ee3e8f5327 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:49.008465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d06373-10f2-42ef-8df3-a7ee3e8f5327 admin admin] PATH_INFO: `/v3/projects/91d9d6efb39a4cb1b8d708fb2f20c580/users/d737341ea53e4a4d8c640e1d46dc6489/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:49.022908 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d06373-10f2-42ef-8df3-a7ee3e8f5327 admin admin] RBAC: Authorizing `identity:create_grant(project_id=91d9d6efb39a4cb1b8d708fb2f20c580, user_id=d737341ea53e4a4d8c640e1d46dc6489, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:49.025541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d06373-10f2-42ef-8df3-a7ee3e8f5327 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:49.124817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1328/2647] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:53:48 2022] PUT /identity/v3/projects/91d9d6efb39a4cb1b8d708fb2f20c580/users/d737341ea53e4a4d8c640e1d46dc6489/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:53:49.136188 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f197cee7-22be-45e9-9fd9-99d55e191dce None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:49.136468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f197cee7-22be-45e9-9fd9-99d55e191dce None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:49.136743 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f197cee7-22be-45e9-9fd9-99d55e191dce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:49.164594 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f197cee7-22be-45e9-9fd9-99d55e191dce None None] MFA Rules not processed for user `d737341ea53e4a4d8c640e1d46dc6489`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:49.165794 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f197cee7-22be-45e9-9fd9-99d55e191dce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:49.170650 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:49.231014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1320/2648] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:53:49 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:53:49.235578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d06373-10f2-42ef-8df3-a7ee3e8f5327 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:49.298915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d06373-10f2-42ef-8df3-a7ee3e8f5327 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:49.303448 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b19a3e7-d69d-4aea-90ef-b1168c64e6d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:49.304038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b19a3e7-d69d-4aea-90ef-b1168c64e6d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:49.304314 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b19a3e7-d69d-4aea-90ef-b1168c64e6d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:49.304582 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b19a3e7-d69d-4aea-90ef-b1168c64e6d1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:49.305192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b19a3e7-d69d-4aea-90ef-b1168c64e6d1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:49.307118 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b19a3e7-d69d-4aea-90ef-b1168c64e6d1 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:49.385195 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1329/2649] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:53:49 2022] POST /identity/v3/projects => generated 379 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:53:49.389215 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f197cee7-22be-45e9-9fd9-99d55e191dce None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:49.405796 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f197cee7-22be-45e9-9fd9-99d55e191dce None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:49.461182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8295310-fbf9-498a-ad6a-49602210df14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:49.461844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8295310-fbf9-498a-ad6a-49602210df14 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:49.462055 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8295310-fbf9-498a-ad6a-49602210df14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:49.462328 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8295310-fbf9-498a-ad6a-49602210df14 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:49.462978 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8295310-fbf9-498a-ad6a-49602210df14 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:49.464911 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8295310-fbf9-498a-ad6a-49602210df14 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:49.572447 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1321/2650] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:53:49 2022] POST /identity/v3/users => generated 290 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:53:49.576653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b19a3e7-d69d-4aea-90ef-b1168c64e6d1 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:49.587302 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b19a3e7-d69d-4aea-90ef-b1168c64e6d1 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:49.591861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b33d87-f8f4-43d5-baf0-801c31413c04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:49.592446 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b33d87-f8f4-43d5-baf0-801c31413c04 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:49.592723 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b33d87-f8f4-43d5-baf0-801c31413c04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:49.592993 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b33d87-f8f4-43d5-baf0-801c31413c04 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:49.593618 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b33d87-f8f4-43d5-baf0-801c31413c04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:49.595249 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b33d87-f8f4-43d5-baf0-801c31413c04 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:49.606680 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1330/2651] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:53:49 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:53:49.610468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8295310-fbf9-498a-ad6a-49602210df14 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:49.620655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8295310-fbf9-498a-ad6a-49602210df14 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:49.625503 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5001d87-a714-4198-89a5-87287e662170 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:49.626105 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5001d87-a714-4198-89a5-87287e662170 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:49.626385 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5001d87-a714-4198-89a5-87287e662170 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:49.626691 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5001d87-a714-4198-89a5-87287e662170 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:49.628269 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5001d87-a714-4198-89a5-87287e662170 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:49.629859 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5001d87-a714-4198-89a5-87287e662170 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:49.644186 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1322/2652] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:53:49 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:53:49.645364 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b33d87-f8f4-43d5-baf0-801c31413c04 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:49.655614 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b33d87-f8f4-43d5-baf0-801c31413c04 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:49.660400 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911b7460-5de3-4f8d-bac4-ba697c75e017 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:49.661113 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911b7460-5de3-4f8d-bac4-ba697c75e017 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:49.661458 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911b7460-5de3-4f8d-bac4-ba697c75e017 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:49.661791 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911b7460-5de3-4f8d-bac4-ba697c75e017 admin admin] PATH_INFO: `/v3/projects/65d7401752b34e23ae3c1960e68c3922/users/2476339d9a2a4af8918e75ad2c60dd4e/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:49.676677 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911b7460-5de3-4f8d-bac4-ba697c75e017 admin admin] RBAC: Authorizing `identity:create_grant(project_id=65d7401752b34e23ae3c1960e68c3922, user_id=2476339d9a2a4af8918e75ad2c60dd4e, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:49.679675 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911b7460-5de3-4f8d-bac4-ba697c75e017 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:49.756928 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1331/2653] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:53:49 2022] PUT /identity/v3/projects/65d7401752b34e23ae3c1960e68c3922/users/2476339d9a2a4af8918e75ad2c60dd4e/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:49.767384 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65caaf38-e44a-4b53-bb80-078b373f888a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:49.771566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65caaf38-e44a-4b53-bb80-078b373f888a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:49.771566 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65caaf38-e44a-4b53-bb80-078b373f888a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:49.796304 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-65caaf38-e44a-4b53-bb80-078b373f888a None None] MFA Rules not processed for user `2476339d9a2a4af8918e75ad2c60dd4e`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:49.800096 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-65caaf38-e44a-4b53-bb80-078b373f888a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:49.861308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1323/2654] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:53:49 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:53:49.865854 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911b7460-5de3-4f8d-bac4-ba697c75e017 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:49.911837 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:49.927833 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911b7460-5de3-4f8d-bac4-ba697c75e017 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:49.932460 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8983709d-b52b-4f54-be16-924610e1b7ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:49.933061 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8983709d-b52b-4f54-be16-924610e1b7ad admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:49.933341 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8983709d-b52b-4f54-be16-924610e1b7ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:49.933614 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8983709d-b52b-4f54-be16-924610e1b7ad admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:49.934243 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8983709d-b52b-4f54-be16-924610e1b7ad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:49.936177 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8983709d-b52b-4f54-be16-924610e1b7ad admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:50.041318 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1332/2655] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:53:49 2022] POST /identity/v3/projects => generated 377 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:53:50.045369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65caaf38-e44a-4b53-bb80-078b373f888a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:50.056467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65caaf38-e44a-4b53-bb80-078b373f888a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:50.068667 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:50.112802 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c178b92-1048-44f5-abb9-c8e78e2a1db6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:50.113390 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c178b92-1048-44f5-abb9-c8e78e2a1db6 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:50.113672 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c178b92-1048-44f5-abb9-c8e78e2a1db6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:50.113947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c178b92-1048-44f5-abb9-c8e78e2a1db6 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:50.114580 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c178b92-1048-44f5-abb9-c8e78e2a1db6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:50.116571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c178b92-1048-44f5-abb9-c8e78e2a1db6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:50.226624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1324/2656] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:53:50 2022] POST /identity/v3/users => generated 295 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:53:50.229419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8983709d-b52b-4f54-be16-924610e1b7ad admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:50.240335 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8983709d-b52b-4f54-be16-924610e1b7ad admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:50.245152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d989543b-9c8c-4b47-9bb6-828b5091472e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:50.245827 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d989543b-9c8c-4b47-9bb6-828b5091472e admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:50.246218 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d989543b-9c8c-4b47-9bb6-828b5091472e admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:50.246496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d989543b-9c8c-4b47-9bb6-828b5091472e admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:50.247188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d989543b-9c8c-4b47-9bb6-828b5091472e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:50.248773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d989543b-9c8c-4b47-9bb6-828b5091472e admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:50.259923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1333/2657] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:53:50 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:53:50.269233 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c178b92-1048-44f5-abb9-c8e78e2a1db6 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:50.279454 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c178b92-1048-44f5-abb9-c8e78e2a1db6 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:50.284027 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab8206d-bd6b-4dea-b939-a56b5397270e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:50.284703 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab8206d-bd6b-4dea-b939-a56b5397270e admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:50.284983 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab8206d-bd6b-4dea-b939-a56b5397270e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:50.285247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab8206d-bd6b-4dea-b939-a56b5397270e admin admin] PATH_INFO: `/v3/projects/e884368231434621add5dc2b643fae9c/users/2ac30ef852354b9d9f1d704f93e5a3bf/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:50.301113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab8206d-bd6b-4dea-b939-a56b5397270e admin admin] RBAC: Authorizing `identity:create_grant(project_id=e884368231434621add5dc2b643fae9c, user_id=2ac30ef852354b9d9f1d704f93e5a3bf, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:50.303879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab8206d-bd6b-4dea-b939-a56b5397270e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:50.371374 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1325/2658] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:53:50 2022] PUT /identity/v3/projects/e884368231434621add5dc2b643fae9c/users/2ac30ef852354b9d9f1d704f93e5a3bf/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:50.378006 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d989543b-9c8c-4b47-9bb6-828b5091472e admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:50.387102 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d989543b-9c8c-4b47-9bb6-828b5091472e admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:50.443786 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdeb88f5-2b6f-4611-a6b8-2261c1bdaa8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:50.444395 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdeb88f5-2b6f-4611-a6b8-2261c1bdaa8a admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:50.444673 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdeb88f5-2b6f-4611-a6b8-2261c1bdaa8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:50.444939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdeb88f5-2b6f-4611-a6b8-2261c1bdaa8a admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:50.445562 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdeb88f5-2b6f-4611-a6b8-2261c1bdaa8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:50.447182 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdeb88f5-2b6f-4611-a6b8-2261c1bdaa8a admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:50.459152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1334/2659] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:53:50 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:53:50.463557 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab8206d-bd6b-4dea-b939-a56b5397270e admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:50.474231 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab8206d-bd6b-4dea-b939-a56b5397270e admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:50.478870 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6bc93c-f5e5-4170-bd18-a598a6fe0edc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:50.479519 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6bc93c-f5e5-4170-bd18-a598a6fe0edc admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:50.479798 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6bc93c-f5e5-4170-bd18-a598a6fe0edc admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:50.480070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6bc93c-f5e5-4170-bd18-a598a6fe0edc admin admin] PATH_INFO: `/v3/projects/e884368231434621add5dc2b643fae9c/users/2ac30ef852354b9d9f1d704f93e5a3bf/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:50.481953 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6bc93c-f5e5-4170-bd18-a598a6fe0edc admin admin] RBAC: Authorizing `identity:create_grant(project_id=e884368231434621add5dc2b643fae9c, user_id=2ac30ef852354b9d9f1d704f93e5a3bf, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:50.484590 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6bc93c-f5e5-4170-bd18-a598a6fe0edc admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:50.495259 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1326/2660] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:53:50 2022] PUT /identity/v3/projects/e884368231434621add5dc2b643fae9c/users/2ac30ef852354b9d9f1d704f93e5a3bf/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:50.506394 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2cd421-f45d-4845-a03b-3cbabd2045f7 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:50.506739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2cd421-f45d-4845-a03b-3cbabd2045f7 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:50.507017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2cd421-f45d-4845-a03b-3cbabd2045f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:50.535108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-2e2cd421-f45d-4845-a03b-3cbabd2045f7 None None] MFA Rules not processed for user `2ac30ef852354b9d9f1d704f93e5a3bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:50.536324 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-2e2cd421-f45d-4845-a03b-3cbabd2045f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:50.602070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1335/2661] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:53:50 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:53:50.613877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6bc93c-f5e5-4170-bd18-a598a6fe0edc admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:50.615603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ee6bc93c-f5e5-4170-bd18-a598a6fe0edc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:50.672518 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6bc93c-f5e5-4170-bd18-a598a6fe0edc admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:50.676747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6499b3f-c2ff-485c-a0a8-64415b120a83 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:50.677359 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6499b3f-c2ff-485c-a0a8-64415b120a83 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:50.677638 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6499b3f-c2ff-485c-a0a8-64415b120a83 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:50.677910 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6499b3f-c2ff-485c-a0a8-64415b120a83 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:50.687464 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6499b3f-c2ff-485c-a0a8-64415b120a83 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:50.688913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6499b3f-c2ff-485c-a0a8-64415b120a83 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:50.698464 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1327/2662] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:53:50 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:53:50.806240 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2cd421-f45d-4845-a03b-3cbabd2045f7 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:50.807747 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-2e2cd421-f45d-4845-a03b-3cbabd2045f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:50.819554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2cd421-f45d-4845-a03b-3cbabd2045f7 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:50.823257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3c77d6-58aa-4e63-8fd9-cd26e17e3827 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:50.823922 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3c77d6-58aa-4e63-8fd9-cd26e17e3827 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:50.824278 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3c77d6-58aa-4e63-8fd9-cd26e17e3827 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:50.824559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3c77d6-58aa-4e63-8fd9-cd26e17e3827 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:50.867173 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:50.885066 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3c77d6-58aa-4e63-8fd9-cd26e17e3827 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:50.886660 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3c77d6-58aa-4e63-8fd9-cd26e17e3827 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:50.896301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1336/2663] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:53:50 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:53:52.489569 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bed7f1e-9157-4628-ae17-40f2cb1c4528 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:52.490141 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bed7f1e-9157-4628-ae17-40f2cb1c4528 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:52.490259 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bed7f1e-9157-4628-ae17-40f2cb1c4528 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:52.491631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1328/2664] 10.222.0.39 () {54 vars in 921 bytes} [Fri Sep 30 09:53:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:53:52.498331 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdf9cdd-236f-456e-b634-d5d81f15c737 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:52.498855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdf9cdd-236f-456e-b634-d5d81f15c737 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:52.498956 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdf9cdd-236f-456e-b634-d5d81f15c737 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:52.520163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-ecdf9cdd-236f-456e-b634-d5d81f15c737 None None] MFA Rules not processed for user `7ff68fb87700423ab53c90520a6bd3b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:52.521460 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ecdf9cdd-236f-456e-b634-d5d81f15c737 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:52.590623 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1337/2665] 10.222.0.39 () {58 vars in 1018 bytes} [Fri Sep 30 09:53:52 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:53:52.602444 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bed7f1e-9157-4628-ae17-40f2cb1c4528 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:52.604176 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8bed7f1e-9157-4628-ae17-40f2cb1c4528 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:52.660688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bed7f1e-9157-4628-ae17-40f2cb1c4528 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:52.664098 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1a8ab7-8058-4232-bd10-16752baa8b50 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf907d4033064fb8a2707b1bbb0a180b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:52.664696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1a8ab7-8058-4232-bd10-16752baa8b50 service glance] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:52.664975 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1a8ab7-8058-4232-bd10-16752baa8b50 service glance] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:52.665254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1a8ab7-8058-4232-bd10-16752baa8b50 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:52.675300 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1a8ab7-8058-4232-bd10-16752baa8b50 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:52.676798 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1a8ab7-8058-4232-bd10-16752baa8b50 service glance] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:52.687687 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1329/2666] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:53:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:53:52.906078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf4435a-7f78-4d61-819d-d5dc7458ae8b None None] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:52.906466 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf4435a-7f78-4d61-819d-d5dc7458ae8b None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:52.906777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf4435a-7f78-4d61-819d-d5dc7458ae8b None None] PATH_INFO: `/` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:52.908142 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1338/2667] 10.222.0.39 () {54 vars in 921 bytes} [Fri Sep 30 09:53:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:53:52.913022 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5ca002-e965-41de-aba0-e42c5189bc19 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:52.913257 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5ca002-e965-41de-aba0-e42c5189bc19 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:52.913535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5ca002-e965-41de-aba0-e42c5189bc19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:52.942381 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:52.957665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-cf5ca002-e965-41de-aba0-e42c5189bc19 None None] MFA Rules not processed for user `9266babd21414ae9998716e5a0478e9d`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:52.959105 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-cf5ca002-e965-41de-aba0-e42c5189bc19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:53.009854 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1330/2668] 10.222.0.39 () {58 vars in 1018 bytes} [Fri Sep 30 09:53:52 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:53:53.022547 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf4435a-7f78-4d61-819d-d5dc7458ae8b None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:53.026343 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0cf4435a-7f78-4d61-819d-d5dc7458ae8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:53.038372 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf4435a-7f78-4d61-819d-d5dc7458ae8b None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:53.042040 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1b8567-e42d-44c7-a3dd-49de8cfa2f5b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:53.042665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1b8567-e42d-44c7-a3dd-49de8cfa2f5b service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:53.042947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1b8567-e42d-44c7-a3dd-49de8cfa2f5b service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:53.043219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1b8567-e42d-44c7-a3dd-49de8cfa2f5b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:53.050780 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1b8567-e42d-44c7-a3dd-49de8cfa2f5b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:53.052195 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1b8567-e42d-44c7-a3dd-49de8cfa2f5b service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:53.059862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1339/2669] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:53:53 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:53:53.144640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d05464-4ce3-46f2-bf63-26c58a81f20d None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:53.144931 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d05464-4ce3-46f2-bf63-26c58a81f20d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:53.145205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d05464-4ce3-46f2-bf63-26c58a81f20d None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:53.146568 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1331/2670] 10.222.0.39 () {54 vars in 921 bytes} [Fri Sep 30 09:53:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:53:53.152029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787a0687-b03e-4e71-b79f-fb9850a66c69 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:53.152423 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787a0687-b03e-4e71-b79f-fb9850a66c69 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:53.152623 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787a0687-b03e-4e71-b79f-fb9850a66c69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:53.173109 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-787a0687-b03e-4e71-b79f-fb9850a66c69 None None] MFA Rules not processed for user `7ff68fb87700423ab53c90520a6bd3b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:53.174787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-787a0687-b03e-4e71-b79f-fb9850a66c69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:53.184240 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1340/2671] 10.222.0.39 () {58 vars in 1018 bytes} [Fri Sep 30 09:53:53 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:53:53.195422 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d05464-4ce3-46f2-bf63-26c58a81f20d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:53.206729 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d05464-4ce3-46f2-bf63-26c58a81f20d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:53.210278 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecff2662-2fd3-4acd-a512-da31abd78b46 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:53.210923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecff2662-2fd3-4acd-a512-da31abd78b46 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:53.211200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecff2662-2fd3-4acd-a512-da31abd78b46 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:53.211472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecff2662-2fd3-4acd-a512-da31abd78b46 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:53.221472 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecff2662-2fd3-4acd-a512-da31abd78b46 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:53.223021 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecff2662-2fd3-4acd-a512-da31abd78b46 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:53.233550 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1332/2672] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:53:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:53:54.886163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787a0687-b03e-4e71-b79f-fb9850a66c69 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:54.951852 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787a0687-b03e-4e71-b79f-fb9850a66c69 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:54.956506 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f692655-8750-4006-864b-2c5b77ccdd1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:54.957137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f692655-8750-4006-864b-2c5b77ccdd1d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:54.957415 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f692655-8750-4006-864b-2c5b77ccdd1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:54.957679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f692655-8750-4006-864b-2c5b77ccdd1d admin admin] PATH_INFO: `/v3/users/d737341ea53e4a4d8c640e1d46dc6489` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:54.958904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f692655-8750-4006-864b-2c5b77ccdd1d admin admin] RBAC: Authorizing `identity:delete_user(user_id=d737341ea53e4a4d8c640e1d46dc6489)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:54.960921 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f692655-8750-4006-864b-2c5b77ccdd1d admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:55.146698 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6f692655-8750-4006-864b-2c5b77ccdd1d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd737341ea53e4a4d8c640e1d46dc6489'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:55.151148 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:55.159085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6f692655-8750-4006-864b-2c5b77ccdd1d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd737341ea53e4a4d8c640e1d46dc6489'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:55.159773 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6f692655-8750-4006-864b-2c5b77ccdd1d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd737341ea53e4a4d8c640e1d46dc6489'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:55.230999 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6f692655-8750-4006-864b-2c5b77ccdd1d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd737341ea53e4a4d8c640e1d46dc6489'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:55.239308 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-6f692655-8750-4006-864b-2c5b77ccdd1d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd737341ea53e4a4d8c640e1d46dc6489'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:55.245401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1341/2673] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:53:54 2022] DELETE /identity/v3/users/d737341ea53e4a4d8c640e1d46dc6489 => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:55.249640 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca66c2b-67a5-47b8-8dca-06cc2e9242f1 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:55.250123 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca66c2b-67a5-47b8-8dca-06cc2e9242f1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:55.250313 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca66c2b-67a5-47b8-8dca-06cc2e9242f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:55.271007 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-4ca66c2b-67a5-47b8-8dca-06cc2e9242f1 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:55.272221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4ca66c2b-67a5-47b8-8dca-06cc2e9242f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:55.331641 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1333/2674] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:53:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:53:55.341392 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f692655-8750-4006-864b-2c5b77ccdd1d admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:55.342882 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-6f692655-8750-4006-864b-2c5b77ccdd1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:55.400872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f692655-8750-4006-864b-2c5b77ccdd1d admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:55.404841 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117b887d-bae6-4285-82dd-6d79f512b08f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:55.405450 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117b887d-bae6-4285-82dd-6d79f512b08f service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:55.405731 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117b887d-bae6-4285-82dd-6d79f512b08f service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:55.406099 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117b887d-bae6-4285-82dd-6d79f512b08f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:55.416102 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117b887d-bae6-4285-82dd-6d79f512b08f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:55.418355 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117b887d-bae6-4285-82dd-6d79f512b08f service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:55.427374 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1342/2675] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:53:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:53:55.877586 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca66c2b-67a5-47b8-8dca-06cc2e9242f1 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:55.879311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4ca66c2b-67a5-47b8-8dca-06cc2e9242f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:55.892443 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca66c2b-67a5-47b8-8dca-06cc2e9242f1 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:55.898430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6719e778-8e67-4a32-9cb5-003e76116ba0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:55.902014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6719e778-8e67-4a32-9cb5-003e76116ba0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:55.902014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6719e778-8e67-4a32-9cb5-003e76116ba0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:55.902014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6719e778-8e67-4a32-9cb5-003e76116ba0 admin admin] PATH_INFO: `/v3/projects/91d9d6efb39a4cb1b8d708fb2f20c580` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:55.902014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6719e778-8e67-4a32-9cb5-003e76116ba0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=91d9d6efb39a4cb1b8d708fb2f20c580)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:55.902393 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6719e778-8e67-4a32-9cb5-003e76116ba0 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:56.035593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6719e778-8e67-4a32-9cb5-003e76116ba0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '91d9d6efb39a4cb1b8d708fb2f20c580', 'request_id': 'req-6719e778-8e67-4a32-9cb5-003e76116ba0'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:56.041253 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6719e778-8e67-4a32-9cb5-003e76116ba0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '91d9d6efb39a4cb1b8d708fb2f20c580', 'request_id': 'req-6719e778-8e67-4a32-9cb5-003e76116ba0'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:56.047559 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6719e778-8e67-4a32-9cb5-003e76116ba0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '91d9d6efb39a4cb1b8d708fb2f20c580', 'request_id': 'req-6719e778-8e67-4a32-9cb5-003e76116ba0'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:56.114117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6719e778-8e67-4a32-9cb5-003e76116ba0 admin admin] The token cache is being invalidate because project 91d9d6efb39a4cb1b8d708fb2f20c580 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:53:56.118514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6719e778-8e67-4a32-9cb5-003e76116ba0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:56.118514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6719e778-8e67-4a32-9cb5-003e76116ba0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:56.118514 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1334/2676] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:53:55 2022] DELETE /identity/v3/projects/91d9d6efb39a4cb1b8d708fb2f20c580 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:56.120648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117b887d-bae6-4285-82dd-6d79f512b08f service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:56.131821 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117b887d-bae6-4285-82dd-6d79f512b08f service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:56.198994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1e5b5d-174f-40e4-885d-8052942d945f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:56.199692 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1e5b5d-174f-40e4-885d-8052942d945f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:56.199843 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1e5b5d-174f-40e4-885d-8052942d945f admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:56.200483 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1e5b5d-174f-40e4-885d-8052942d945f admin admin] PATH_INFO: `/v3/users/2476339d9a2a4af8918e75ad2c60dd4e` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:56.201257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1e5b5d-174f-40e4-885d-8052942d945f admin admin] RBAC: Authorizing `identity:delete_user(user_id=2476339d9a2a4af8918e75ad2c60dd4e)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:56.203350 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1e5b5d-174f-40e4-885d-8052942d945f admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:56.386239 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7c1e5b5d-174f-40e4-885d-8052942d945f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2476339d9a2a4af8918e75ad2c60dd4e'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:56.398898 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7c1e5b5d-174f-40e4-885d-8052942d945f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2476339d9a2a4af8918e75ad2c60dd4e'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:56.399637 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7c1e5b5d-174f-40e4-885d-8052942d945f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2476339d9a2a4af8918e75ad2c60dd4e'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:56.462493 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7c1e5b5d-174f-40e4-885d-8052942d945f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2476339d9a2a4af8918e75ad2c60dd4e'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:56.464889 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:56.471393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7c1e5b5d-174f-40e4-885d-8052942d945f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2476339d9a2a4af8918e75ad2c60dd4e'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:56.477522 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1343/2677] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:53:56 2022] DELETE /identity/v3/users/2476339d9a2a4af8918e75ad2c60dd4e => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:56.928786 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6719e778-8e67-4a32-9cb5-003e76116ba0 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:56.930229 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6719e778-8e67-4a32-9cb5-003e76116ba0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:56.943486 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6719e778-8e67-4a32-9cb5-003e76116ba0 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:56.962194 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:57.003333 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519a4123-4ae0-4013-a67f-71880e34a601 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:57.004006 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519a4123-4ae0-4013-a67f-71880e34a601 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:57.004281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519a4123-4ae0-4013-a67f-71880e34a601 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:57.004652 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519a4123-4ae0-4013-a67f-71880e34a601 admin admin] PATH_INFO: `/v3/projects/65d7401752b34e23ae3c1960e68c3922` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:57.005843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519a4123-4ae0-4013-a67f-71880e34a601 admin admin] RBAC: Authorizing `identity:delete_project(project_id=65d7401752b34e23ae3c1960e68c3922)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:57.007718 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519a4123-4ae0-4013-a67f-71880e34a601 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:57.059812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0c745e-5884-49d8-a10e-c0e229b8bfe2 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:57.060107 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0c745e-5884-49d8-a10e-c0e229b8bfe2 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:57.060382 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0c745e-5884-49d8-a10e-c0e229b8bfe2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:57.079498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-0b0c745e-5884-49d8-a10e-c0e229b8bfe2 None None] MFA Rules not processed for user `7d6e0545a40a436d8c63820b795551a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:57.081264 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0b0c745e-5884-49d8-a10e-c0e229b8bfe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:57.140082 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-519a4123-4ae0-4013-a67f-71880e34a601 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '65d7401752b34e23ae3c1960e68c3922', 'request_id': 'req-519a4123-4ae0-4013-a67f-71880e34a601'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:57.145539 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1344/2678] 10.222.0.39 () {58 vars in 1009 bytes} [Fri Sep 30 09:53:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:53:57.146818 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-519a4123-4ae0-4013-a67f-71880e34a601 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '65d7401752b34e23ae3c1960e68c3922', 'request_id': 'req-519a4123-4ae0-4013-a67f-71880e34a601'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:57.153314 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-519a4123-4ae0-4013-a67f-71880e34a601 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '65d7401752b34e23ae3c1960e68c3922', 'request_id': 'req-519a4123-4ae0-4013-a67f-71880e34a601'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:57.161135 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0c745e-5884-49d8-a10e-c0e229b8bfe2 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:57.162704 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0b0c745e-5884-49d8-a10e-c0e229b8bfe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:57.230642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-519a4123-4ae0-4013-a67f-71880e34a601 admin admin] The token cache is being invalidate because project 65d7401752b34e23ae3c1960e68c3922 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:53:57.231187 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-519a4123-4ae0-4013-a67f-71880e34a601 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:57.231919 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-519a4123-4ae0-4013-a67f-71880e34a601 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:57.234638 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1335/2679] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:53:56 2022] DELETE /identity/v3/projects/65d7401752b34e23ae3c1960e68c3922 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:57.237480 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519a4123-4ae0-4013-a67f-71880e34a601 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:57.239806 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-519a4123-4ae0-4013-a67f-71880e34a601 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:57.251747 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0c745e-5884-49d8-a10e-c0e229b8bfe2 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:57.319716 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519a4123-4ae0-4013-a67f-71880e34a601 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:57.320319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1fc640-cec3-45f8-8d82-ef95c8f58ab9 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e5ad921e6fa41c297696ec7279f0fc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:57.320763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1fc640-cec3-45f8-8d82-ef95c8f58ab9 service swift] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:57.321115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1fc640-cec3-45f8-8d82-ef95c8f58ab9 service swift] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:57.321321 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1fc640-cec3-45f8-8d82-ef95c8f58ab9 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:57.323104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-6a1fc640-cec3-45f8-8d82-ef95c8f58ab9 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:57.332818 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188ad691-12d4-450c-8f27-60a049b09fe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:57.333668 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188ad691-12d4-450c-8f27-60a049b09fe6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:57.334226 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188ad691-12d4-450c-8f27-60a049b09fe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:57.334528 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188ad691-12d4-450c-8f27-60a049b09fe6 admin admin] PATH_INFO: `/v3/users/2ac30ef852354b9d9f1d704f93e5a3bf` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:57.335710 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188ad691-12d4-450c-8f27-60a049b09fe6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ac30ef852354b9d9f1d704f93e5a3bf)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:57.337896 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188ad691-12d4-450c-8f27-60a049b09fe6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:57.357228 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:57.382224 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1fc640-cec3-45f8-8d82-ef95c8f58ab9 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:57.383859 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1fc640-cec3-45f8-8d82-ef95c8f58ab9 service swift] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:57.392093 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1345/2680] 10.222.0.39 () {60 vars in 1384 bytes} [Fri Sep 30 09:53:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 231 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 30 09:53:57.554401 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-188ad691-12d4-450c-8f27-60a049b09fe6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ac30ef852354b9d9f1d704f93e5a3bf'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:57.563023 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-188ad691-12d4-450c-8f27-60a049b09fe6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ac30ef852354b9d9f1d704f93e5a3bf'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:57.563023 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-188ad691-12d4-450c-8f27-60a049b09fe6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ac30ef852354b9d9f1d704f93e5a3bf'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:57.563023 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:57.565654 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-188ad691-12d4-450c-8f27-60a049b09fe6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ac30ef852354b9d9f1d704f93e5a3bf'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:57.670350 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-188ad691-12d4-450c-8f27-60a049b09fe6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ac30ef852354b9d9f1d704f93e5a3bf'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:57.703037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1336/2681] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:53:57 2022] DELETE /identity/v3/users/2ac30ef852354b9d9f1d704f93e5a3bf => generated 0 bytes in 462 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:53:58.165972 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1fc640-cec3-45f8-8d82-ef95c8f58ab9 service swift] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:58.184459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1fc640-cec3-45f8-8d82-ef95c8f58ab9 service swift] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:58.255958 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f34cfdf-92ec-4d38-b273-b5167ec59212 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:58.256825 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f34cfdf-92ec-4d38-b273-b5167ec59212 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:58.257165 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f34cfdf-92ec-4d38-b273-b5167ec59212 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:58.257570 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f34cfdf-92ec-4d38-b273-b5167ec59212 admin admin] PATH_INFO: `/v3/projects/e884368231434621add5dc2b643fae9c` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:58.258964 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f34cfdf-92ec-4d38-b273-b5167ec59212 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e884368231434621add5dc2b643fae9c)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:58.261042 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f34cfdf-92ec-4d38-b273-b5167ec59212 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:58.367776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-3f34cfdf-92ec-4d38-b273-b5167ec59212 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e884368231434621add5dc2b643fae9c', 'request_id': 'req-3f34cfdf-92ec-4d38-b273-b5167ec59212'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:58.375374 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-3f34cfdf-92ec-4d38-b273-b5167ec59212 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e884368231434621add5dc2b643fae9c', 'request_id': 'req-3f34cfdf-92ec-4d38-b273-b5167ec59212'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:58.383215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-3f34cfdf-92ec-4d38-b273-b5167ec59212 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e884368231434621add5dc2b643fae9c', 'request_id': 'req-3f34cfdf-92ec-4d38-b273-b5167ec59212'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:58.431112 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-3f34cfdf-92ec-4d38-b273-b5167ec59212 admin admin] The token cache is being invalidate because project e884368231434621add5dc2b643fae9c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:53:58.431487 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-3f34cfdf-92ec-4d38-b273-b5167ec59212 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:58.432238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-3f34cfdf-92ec-4d38-b273-b5167ec59212 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:53:58.439000 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1346/2682] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:53:58 2022] DELETE /identity/v3/projects/e884368231434621add5dc2b643fae9c => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:58.440319 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81043334-d76e-4858-9cc2-cc1f5830e20e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:58.440811 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81043334-d76e-4858-9cc2-cc1f5830e20e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:58.440913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81043334-d76e-4858-9cc2-cc1f5830e20e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:58.488040 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-81043334-d76e-4858-9cc2-cc1f5830e20e None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:58.489446 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-81043334-d76e-4858-9cc2-cc1f5830e20e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:58.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1337/2683] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:53:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:53:58.601507 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113fd698-469d-4c3f-b9a4-a6dbf623ef65 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:58.601799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113fd698-469d-4c3f-b9a4-a6dbf623ef65 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:58.602044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113fd698-469d-4c3f-b9a4-a6dbf623ef65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:58.623689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-113fd698-469d-4c3f-b9a4-a6dbf623ef65 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:58.624912 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-113fd698-469d-4c3f-b9a4-a6dbf623ef65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:58.633757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1347/2684] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:53:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:53:58.637961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81043334-d76e-4858-9cc2-cc1f5830e20e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:58.649348 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81043334-d76e-4858-9cc2-cc1f5830e20e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:58.654193 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f854b944-44d2-4261-9ea5-0e5781f29809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:58.654976 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f854b944-44d2-4261-9ea5-0e5781f29809 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:58.655296 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f854b944-44d2-4261-9ea5-0e5781f29809 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:58.655570 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f854b944-44d2-4261-9ea5-0e5781f29809 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:58.656224 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f854b944-44d2-4261-9ea5-0e5781f29809 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:53:58.656522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f854b944-44d2-4261-9ea5-0e5781f29809 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:58.658156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f854b944-44d2-4261-9ea5-0e5781f29809 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:58.670708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1338/2685] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:53:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:53:58.675147 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113fd698-469d-4c3f-b9a4-a6dbf623ef65 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:58.685491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113fd698-469d-4c3f-b9a4-a6dbf623ef65 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:58.690913 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d78809-2630-405b-bd7f-015985776ab1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:58.691517 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d78809-2630-405b-bd7f-015985776ab1 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:58.691806 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d78809-2630-405b-bd7f-015985776ab1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:58.692059 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d78809-2630-405b-bd7f-015985776ab1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:58.692682 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d78809-2630-405b-bd7f-015985776ab1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:58.694751 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d78809-2630-405b-bd7f-015985776ab1 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:58.763849 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1348/2686] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:53:58 2022] POST /identity/v3/projects => generated 373 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:53:58.768138 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f854b944-44d2-4261-9ea5-0e5781f29809 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:58.781960 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f854b944-44d2-4261-9ea5-0e5781f29809 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:58.873521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b628af-c8e2-4ee8-8252-2b0b4a7b7d0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:58.874137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b628af-c8e2-4ee8-8252-2b0b4a7b7d0d admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:58.874414 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b628af-c8e2-4ee8-8252-2b0b4a7b7d0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:58.874714 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b628af-c8e2-4ee8-8252-2b0b4a7b7d0d admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:58.875360 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b628af-c8e2-4ee8-8252-2b0b4a7b7d0d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:58.877393 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b628af-c8e2-4ee8-8252-2b0b4a7b7d0d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:58.942519 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1339/2687] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:53:58 2022] POST /identity/v3/users => generated 287 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:53:58.943775 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d78809-2630-405b-bd7f-015985776ab1 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:58.954635 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d78809-2630-405b-bd7f-015985776ab1 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:58.959479 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078063a5-567d-42f8-a383-e9e638033651 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:58.960169 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078063a5-567d-42f8-a383-e9e638033651 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:58.960453 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078063a5-567d-42f8-a383-e9e638033651 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:58.960727 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078063a5-567d-42f8-a383-e9e638033651 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:58.961600 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078063a5-567d-42f8-a383-e9e638033651 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:58.963269 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078063a5-567d-42f8-a383-e9e638033651 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:58.975295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1349/2688] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:53:58 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:53:58.979696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b628af-c8e2-4ee8-8252-2b0b4a7b7d0d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:58.990214 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b628af-c8e2-4ee8-8252-2b0b4a7b7d0d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:58.994954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6790dfa-f00a-41fc-abb9-b5891cf302ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:58.995565 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6790dfa-f00a-41fc-abb9-b5891cf302ac admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:58.995883 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6790dfa-f00a-41fc-abb9-b5891cf302ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:58.996296 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6790dfa-f00a-41fc-abb9-b5891cf302ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:58.996940 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6790dfa-f00a-41fc-abb9-b5891cf302ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:58.998542 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6790dfa-f00a-41fc-abb9-b5891cf302ac admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:59.017113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1340/2689] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:53:58 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:53:59.021855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078063a5-567d-42f8-a383-e9e638033651 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:59.032001 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078063a5-567d-42f8-a383-e9e638033651 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:59.036881 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693acdac-6062-46ed-a602-8d1ced9c908b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:59.037567 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693acdac-6062-46ed-a602-8d1ced9c908b admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:59.037816 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693acdac-6062-46ed-a602-8d1ced9c908b admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:59.038110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693acdac-6062-46ed-a602-8d1ced9c908b admin admin] PATH_INFO: `/v3/projects/d64c50fedbf0431f84ad9ac73c3c5506/users/390b054d9e7f4d6d9ed51656aff5d133/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:59.056076 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693acdac-6062-46ed-a602-8d1ced9c908b admin admin] RBAC: Authorizing `identity:create_grant(project_id=d64c50fedbf0431f84ad9ac73c3c5506, user_id=390b054d9e7f4d6d9ed51656aff5d133, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:59.058783 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693acdac-6062-46ed-a602-8d1ced9c908b admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:59.117881 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:59.123238 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1350/2690] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:53:59 2022] PUT /identity/v3/projects/d64c50fedbf0431f84ad9ac73c3c5506/users/390b054d9e7f4d6d9ed51656aff5d133/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:53:59.129386 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3f7ee3-e145-4514-9592-f5230e3e2b9e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:59.129735 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3f7ee3-e145-4514-9592-f5230e3e2b9e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:59.130080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3f7ee3-e145-4514-9592-f5230e3e2b9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:59.153863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ee3f7ee3-e145-4514-9592-f5230e3e2b9e None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:53:59.155435 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ee3f7ee3-e145-4514-9592-f5230e3e2b9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:59.170890 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:53:59.224217 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1341/2691] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:53:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:53:59.234106 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693acdac-6062-46ed-a602-8d1ced9c908b admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:53:59.236705 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-693acdac-6062-46ed-a602-8d1ced9c908b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:53:59.297764 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693acdac-6062-46ed-a602-8d1ced9c908b admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:53:59.302084 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149a2135-02ab-4088-b093-76ec338f8df9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:53:59.302743 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149a2135-02ab-4088-b093-76ec338f8df9 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:53:59.303021 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149a2135-02ab-4088-b093-76ec338f8df9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:53:59.303288 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149a2135-02ab-4088-b093-76ec338f8df9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:53:59.315845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149a2135-02ab-4088-b093-76ec338f8df9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:53:59.317363 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149a2135-02ab-4088-b093-76ec338f8df9 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:53:59.327585 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1351/2692] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:53:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:54:01.072143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5898445-4801-4442-b2c4-fb0b0068b411 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:01.072627 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5898445-4801-4442-b2c4-fb0b0068b411 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:01.072731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5898445-4801-4442-b2c4-fb0b0068b411 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:01.109522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-a5898445-4801-4442-b2c4-fb0b0068b411 None None] MFA Rules not processed for user `390b054d9e7f4d6d9ed51656aff5d133`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:54:01.110736 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a5898445-4801-4442-b2c4-fb0b0068b411 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:01.206665 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1342/2693] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:54:01 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:54:01.210351 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149a2135-02ab-4088-b093-76ec338f8df9 service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:01.225062 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149a2135-02ab-4088-b093-76ec338f8df9 service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:01.229959 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fea4ac-2ca4-49f1-8241-b1144fb386c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:01.230574 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fea4ac-2ca4-49f1-8241-b1144fb386c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:01.230887 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fea4ac-2ca4-49f1-8241-b1144fb386c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:01.231354 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fea4ac-2ca4-49f1-8241-b1144fb386c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:01.231825 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99fea4ac-2ca4-49f1-8241-b1144fb386c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:01.233769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99fea4ac-2ca4-49f1-8241-b1144fb386c6 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:01.324159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1352/2694] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:54:01 2022] POST /identity/v3/projects => generated 371 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:54:01.328402 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5898445-4801-4442-b2c4-fb0b0068b411 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:01.342785 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5898445-4801-4442-b2c4-fb0b0068b411 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:01.405739 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49183851-9c4c-4170-ac6f-702c692fec8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:01.406369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49183851-9c4c-4170-ac6f-702c692fec8f admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:01.406687 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49183851-9c4c-4170-ac6f-702c692fec8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:01.407018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49183851-9c4c-4170-ac6f-702c692fec8f admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:01.407672 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49183851-9c4c-4170-ac6f-702c692fec8f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:01.410180 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49183851-9c4c-4170-ac6f-702c692fec8f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:01.494680 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1343/2695] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:54:01 2022] POST /identity/v3/users => generated 292 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:54:01.498614 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fea4ac-2ca4-49f1-8241-b1144fb386c6 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:01.509983 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fea4ac-2ca4-49f1-8241-b1144fb386c6 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:01.514696 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a21d404-5394-451f-a163-593f61bbc8a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:01.515304 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a21d404-5394-451f-a163-593f61bbc8a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:01.515624 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a21d404-5394-451f-a163-593f61bbc8a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:01.518992 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a21d404-5394-451f-a163-593f61bbc8a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:01.518992 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a21d404-5394-451f-a163-593f61bbc8a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:01.518992 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a21d404-5394-451f-a163-593f61bbc8a4 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:01.533780 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1353/2696] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:54:01 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:54:01.534838 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49183851-9c4c-4170-ac6f-702c692fec8f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:01.546664 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49183851-9c4c-4170-ac6f-702c692fec8f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:01.552765 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf76f0e-2086-4d5e-9d9f-7d87b31df332 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:01.553556 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf76f0e-2086-4d5e-9d9f-7d87b31df332 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:01.553909 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf76f0e-2086-4d5e-9d9f-7d87b31df332 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:01.554254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf76f0e-2086-4d5e-9d9f-7d87b31df332 admin admin] PATH_INFO: `/v3/projects/be70ba68d13c4b9bbac7f0d788641990/users/89c82a12dca54a3e96ff6237120c5c4a/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:01.570309 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf76f0e-2086-4d5e-9d9f-7d87b31df332 admin admin] RBAC: Authorizing `identity:create_grant(project_id=be70ba68d13c4b9bbac7f0d788641990, user_id=89c82a12dca54a3e96ff6237120c5c4a, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:01.573023 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf76f0e-2086-4d5e-9d9f-7d87b31df332 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:01.640652 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1344/2697] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:54:01 2022] PUT /identity/v3/projects/be70ba68d13c4b9bbac7f0d788641990/users/89c82a12dca54a3e96ff6237120c5c4a/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:01.645393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a21d404-5394-451f-a163-593f61bbc8a4 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:01.657451 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a21d404-5394-451f-a163-593f61bbc8a4 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:01.718690 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5833897-9f99-4c7d-ad7b-47f795d6af95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:01.719328 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5833897-9f99-4c7d-ad7b-47f795d6af95 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:01.719609 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5833897-9f99-4c7d-ad7b-47f795d6af95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:01.719880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5833897-9f99-4c7d-ad7b-47f795d6af95 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:01.720569 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5833897-9f99-4c7d-ad7b-47f795d6af95 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:01.722172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5833897-9f99-4c7d-ad7b-47f795d6af95 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:01.732820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1354/2698] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:54:01 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:54:01.737578 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf76f0e-2086-4d5e-9d9f-7d87b31df332 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:01.748385 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf76f0e-2086-4d5e-9d9f-7d87b31df332 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:01.753885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a983f6a4-9d41-4c3c-b4ab-b514dbc1c693 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:01.754552 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a983f6a4-9d41-4c3c-b4ab-b514dbc1c693 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:01.754892 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a983f6a4-9d41-4c3c-b4ab-b514dbc1c693 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:01.755159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a983f6a4-9d41-4c3c-b4ab-b514dbc1c693 admin admin] PATH_INFO: `/v3/projects/be70ba68d13c4b9bbac7f0d788641990/users/89c82a12dca54a3e96ff6237120c5c4a/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:01.757217 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a983f6a4-9d41-4c3c-b4ab-b514dbc1c693 admin admin] RBAC: Authorizing `identity:create_grant(project_id=be70ba68d13c4b9bbac7f0d788641990, user_id=89c82a12dca54a3e96ff6237120c5c4a, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:01.760029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a983f6a4-9d41-4c3c-b4ab-b514dbc1c693 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:01.772737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1345/2699] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:54:01 2022] PUT /identity/v3/projects/be70ba68d13c4b9bbac7f0d788641990/users/89c82a12dca54a3e96ff6237120c5c4a/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:01.906167 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5833897-9f99-4c7d-ad7b-47f795d6af95 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:01.998664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5833897-9f99-4c7d-ad7b-47f795d6af95 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:02.019536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:02.053077 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c36f431-b6b2-47d0-aa4b-673b488bbb0b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:02.053756 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c36f431-b6b2-47d0-aa4b-673b488bbb0b service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:02.054117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c36f431-b6b2-47d0-aa4b-673b488bbb0b service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:02.054463 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c36f431-b6b2-47d0-aa4b-673b488bbb0b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:02.063976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-4c36f431-b6b2-47d0-aa4b-673b488bbb0b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:02.179185 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c36f431-b6b2-47d0-aa4b-673b488bbb0b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:02.179185 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c36f431-b6b2-47d0-aa4b-673b488bbb0b service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:02.197982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1355/2700] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:54:01 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 288 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:54:03.870868 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801a1c01-c1c0-4ffe-94ad-4ef69f5e3887 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:03.871509 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801a1c01-c1c0-4ffe-94ad-4ef69f5e3887 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:03.871509 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801a1c01-c1c0-4ffe-94ad-4ef69f5e3887 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:03.880958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:03.920225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-801a1c01-c1c0-4ffe-94ad-4ef69f5e3887 None None] MFA Rules not processed for user `89c82a12dca54a3e96ff6237120c5c4a`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:54:03.920225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-801a1c01-c1c0-4ffe-94ad-4ef69f5e3887 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:04.038749 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1346/2701] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:54:03 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:54:04.061342 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c36f431-b6b2-47d0-aa4b-673b488bbb0b service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:04.067254 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-4c36f431-b6b2-47d0-aa4b-673b488bbb0b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:04.234627 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c36f431-b6b2-47d0-aa4b-673b488bbb0b service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:04.239684 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3313e52-65f4-4076-bfc1-859c2092e49c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:04.240305 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3313e52-65f4-4076-bfc1-859c2092e49c service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:04.244994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3313e52-65f4-4076-bfc1-859c2092e49c service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:04.244994 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3313e52-65f4-4076-bfc1-859c2092e49c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:04.377683 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3313e52-65f4-4076-bfc1-859c2092e49c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:04.379631 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3313e52-65f4-4076-bfc1-859c2092e49c service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:04.392918 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1356/2702] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:54:04 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:54:06.389339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801a1c01-c1c0-4ffe-94ad-4ef69f5e3887 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:06.390951 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-801a1c01-c1c0-4ffe-94ad-4ef69f5e3887 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:06.406198 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801a1c01-c1c0-4ffe-94ad-4ef69f5e3887 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:06.410685 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c35b998-cc19-4c98-a315-0448c9b54b41 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:06.411442 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c35b998-cc19-4c98-a315-0448c9b54b41 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:06.411816 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c35b998-cc19-4c98-a315-0448c9b54b41 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:06.412183 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c35b998-cc19-4c98-a315-0448c9b54b41 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:06.423319 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c35b998-cc19-4c98-a315-0448c9b54b41 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:06.431356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c35b998-cc19-4c98-a315-0448c9b54b41 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:06.454541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1347/2703] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:54:06 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:54:13.314859 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3313e52-65f4-4076-bfc1-859c2092e49c service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:13.316330 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b3313e52-65f4-4076-bfc1-859c2092e49c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:13.328437 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3313e52-65f4-4076-bfc1-859c2092e49c service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:13.332534 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e43d83-3ab6-40a9-bbe1-8d43747ed825 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:13.333129 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e43d83-3ab6-40a9-bbe1-8d43747ed825 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:13.333409 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e43d83-3ab6-40a9-bbe1-8d43747ed825 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:13.333679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e43d83-3ab6-40a9-bbe1-8d43747ed825 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:13.335413 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-84e43d83-3ab6-40a9-bbe1-8d43747ed825 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:13.365579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:13.398351 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e43d83-3ab6-40a9-bbe1-8d43747ed825 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:13.401298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e43d83-3ab6-40a9-bbe1-8d43747ed825 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:13.410078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1357/2704] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:54:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:54:16.438171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=39 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:20.976136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c35b998-cc19-4c98-a315-0448c9b54b41 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:20.977756 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3c35b998-cc19-4c98-a315-0448c9b54b41 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:20.991562 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c35b998-cc19-4c98-a315-0448c9b54b41 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:20.996073 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e968f1-e971-4920-b2c2-ed1b2a80e81f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:20.996731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e968f1-e971-4920-b2c2-ed1b2a80e81f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:20.997007 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e968f1-e971-4920-b2c2-ed1b2a80e81f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:20.997290 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e968f1-e971-4920-b2c2-ed1b2a80e81f admin admin] PATH_INFO: `/v3/users/20bee9f38eb14541a393037eaa389a44` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:20.999053 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e968f1-e971-4920-b2c2-ed1b2a80e81f admin admin] RBAC: Authorizing `identity:delete_user(user_id=20bee9f38eb14541a393037eaa389a44)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:21.001216 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e968f1-e971-4920-b2c2-ed1b2a80e81f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:21.191205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b0e968f1-e971-4920-b2c2-ed1b2a80e81f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '20bee9f38eb14541a393037eaa389a44'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:21.191961 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b0e968f1-e971-4920-b2c2-ed1b2a80e81f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '20bee9f38eb14541a393037eaa389a44'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:21.192633 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b0e968f1-e971-4920-b2c2-ed1b2a80e81f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '20bee9f38eb14541a393037eaa389a44'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:21.201331 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b0e968f1-e971-4920-b2c2-ed1b2a80e81f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '20bee9f38eb14541a393037eaa389a44'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:21.269008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b0e968f1-e971-4920-b2c2-ed1b2a80e81f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '20bee9f38eb14541a393037eaa389a44'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:21.291973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1348/2705] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:54:20 2022] DELETE /identity/v3/users/20bee9f38eb14541a393037eaa389a44 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:21.527240 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e43d83-3ab6-40a9-bbe1-8d43747ed825 service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:21.537436 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e43d83-3ab6-40a9-bbe1-8d43747ed825 service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:21.597397 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a812a7-80cf-4ca2-9e7a-45e5f56094fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:21.598029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a812a7-80cf-4ca2-9e7a-45e5f56094fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:21.598311 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a812a7-80cf-4ca2-9e7a-45e5f56094fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:21.598699 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a812a7-80cf-4ca2-9e7a-45e5f56094fa admin admin] PATH_INFO: `/v3/projects/1330031e214045bf9d45c73da1445255` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:21.599834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a812a7-80cf-4ca2-9e7a-45e5f56094fa admin admin] RBAC: Authorizing `identity:delete_project(project_id=1330031e214045bf9d45c73da1445255)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:21.601738 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a812a7-80cf-4ca2-9e7a-45e5f56094fa admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:21.718415 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-c0a812a7-80cf-4ca2-9e7a-45e5f56094fa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1330031e214045bf9d45c73da1445255', 'request_id': 'req-c0a812a7-80cf-4ca2-9e7a-45e5f56094fa'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:21.718619 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4875e669-f3a5-4c6c-adc7-9e27f3ff18c0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:21.718937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4875e669-f3a5-4c6c-adc7-9e27f3ff18c0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:21.719279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4875e669-f3a5-4c6c-adc7-9e27f3ff18c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:21.724244 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-c0a812a7-80cf-4ca2-9e7a-45e5f56094fa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1330031e214045bf9d45c73da1445255', 'request_id': 'req-c0a812a7-80cf-4ca2-9e7a-45e5f56094fa'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:21.730642 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-c0a812a7-80cf-4ca2-9e7a-45e5f56094fa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1330031e214045bf9d45c73da1445255', 'request_id': 'req-c0a812a7-80cf-4ca2-9e7a-45e5f56094fa'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:21.740336 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-4875e669-f3a5-4c6c-adc7-9e27f3ff18c0 None None] MFA Rules not processed for user `7d6e0545a40a436d8c63820b795551a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:54:21.742161 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4875e669-f3a5-4c6c-adc7-9e27f3ff18c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:21.806189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-c0a812a7-80cf-4ca2-9e7a-45e5f56094fa admin admin] The token cache is being invalidate because project 1330031e214045bf9d45c73da1445255 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:54:21.806189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-c0a812a7-80cf-4ca2-9e7a-45e5f56094fa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:21.806189 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-c0a812a7-80cf-4ca2-9e7a-45e5f56094fa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:21.806492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1349/2707] 10.222.0.39 () {58 vars in 967 bytes} [Fri Sep 30 09:54:21 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:54:21.806817 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1358/2707] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:54:21 2022] DELETE /identity/v3/projects/1330031e214045bf9d45c73da1445255 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:21.810037 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4875e669-f3a5-4c6c-adc7-9e27f3ff18c0 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:21.811652 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4875e669-f3a5-4c6c-adc7-9e27f3ff18c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:21.818192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a812a7-80cf-4ca2-9e7a-45e5f56094fa admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:21.820231 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c0a812a7-80cf-4ca2-9e7a-45e5f56094fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:21.886979 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4875e669-f3a5-4c6c-adc7-9e27f3ff18c0 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:21.889346 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:21.891545 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d39eb3a-9405-48fa-b51b-de1b3320e3c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:21.892180 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d39eb3a-9405-48fa-b51b-de1b3320e3c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:21.892483 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d39eb3a-9405-48fa-b51b-de1b3320e3c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:21.892727 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d39eb3a-9405-48fa-b51b-de1b3320e3c0 admin admin] PATH_INFO: `/v3/users/29a0903dfdb042b8aea469db4fe31599` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:21.893872 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d39eb3a-9405-48fa-b51b-de1b3320e3c0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=29a0903dfdb042b8aea469db4fe31599)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:21.895857 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d39eb3a-9405-48fa-b51b-de1b3320e3c0 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:21.907538 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a812a7-80cf-4ca2-9e7a-45e5f56094fa admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:21.914277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeab886b-a9c1-4505-a538-ed3ee1228a80 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e5ad921e6fa41c297696ec7279f0fc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:21.915617 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeab886b-a9c1-4505-a538-ed3ee1228a80 service swift] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:21.916207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeab886b-a9c1-4505-a538-ed3ee1228a80 service swift] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:21.916553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeab886b-a9c1-4505-a538-ed3ee1228a80 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:21.918607 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-aeab886b-a9c1-4505-a538-ed3ee1228a80 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:21.919068 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:21.932130 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeab886b-a9c1-4505-a538-ed3ee1228a80 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:21.933898 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeab886b-a9c1-4505-a538-ed3ee1228a80 service swift] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:21.942363 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1359/2708] 10.222.0.39 () {60 vars in 1384 bytes} [Fri Sep 30 09:54:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 30 09:54:22.088862 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7d39eb3a-9405-48fa-b51b-de1b3320e3c0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '29a0903dfdb042b8aea469db4fe31599'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:22.099281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7d39eb3a-9405-48fa-b51b-de1b3320e3c0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '29a0903dfdb042b8aea469db4fe31599'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:22.099281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7d39eb3a-9405-48fa-b51b-de1b3320e3c0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '29a0903dfdb042b8aea469db4fe31599'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:22.099281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7d39eb3a-9405-48fa-b51b-de1b3320e3c0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '29a0903dfdb042b8aea469db4fe31599'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:22.188690 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7d39eb3a-9405-48fa-b51b-de1b3320e3c0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '29a0903dfdb042b8aea469db4fe31599'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:22.220491 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1350/2709] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:54:21 2022] DELETE /identity/v3/users/29a0903dfdb042b8aea469db4fe31599 => generated 0 bytes in 411 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:22.661412 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeab886b-a9c1-4505-a538-ed3ee1228a80 service swift] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:22.664430 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-aeab886b-a9c1-4505-a538-ed3ee1228a80 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:22.671058 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d39eb3a-9405-48fa-b51b-de1b3320e3c0 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:22.672556 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7d39eb3a-9405-48fa-b51b-de1b3320e3c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:22.681100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeab886b-a9c1-4505-a538-ed3ee1228a80 service swift] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:22.747737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d39eb3a-9405-48fa-b51b-de1b3320e3c0 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:22.751877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996958f8-b264-4141-990d-a96a36daff98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:22.752828 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53edc006-3812-4870-b2e8-65bcd859e45d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:22.752988 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996958f8-b264-4141-990d-a96a36daff98 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:22.753291 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996958f8-b264-4141-990d-a96a36daff98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:22.753736 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53edc006-3812-4870-b2e8-65bcd859e45d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:22.753982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996958f8-b264-4141-990d-a96a36daff98 admin admin] PATH_INFO: `/v3/projects/82b48c489465430a836de26d4aa39440` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:22.754105 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53edc006-3812-4870-b2e8-65bcd859e45d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:22.754464 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53edc006-3812-4870-b2e8-65bcd859e45d admin admin] PATH_INFO: `/v3/users/390b054d9e7f4d6d9ed51656aff5d133` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:22.755575 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996958f8-b264-4141-990d-a96a36daff98 admin admin] RBAC: Authorizing `identity:delete_project(project_id=82b48c489465430a836de26d4aa39440)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:22.755719 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53edc006-3812-4870-b2e8-65bcd859e45d admin admin] RBAC: Authorizing `identity:delete_user(user_id=390b054d9e7f4d6d9ed51656aff5d133)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:22.757507 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996958f8-b264-4141-990d-a96a36daff98 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:22.757626 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53edc006-3812-4870-b2e8-65bcd859e45d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:22.977906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-53edc006-3812-4870-b2e8-65bcd859e45d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '390b054d9e7f4d6d9ed51656aff5d133'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:22.978728 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-53edc006-3812-4870-b2e8-65bcd859e45d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '390b054d9e7f4d6d9ed51656aff5d133'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:22.979393 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-53edc006-3812-4870-b2e8-65bcd859e45d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '390b054d9e7f4d6d9ed51656aff5d133'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:22.979623 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-996958f8-b264-4141-990d-a96a36daff98 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '82b48c489465430a836de26d4aa39440', 'request_id': 'req-996958f8-b264-4141-990d-a96a36daff98'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:22.986656 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-996958f8-b264-4141-990d-a96a36daff98 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '82b48c489465430a836de26d4aa39440', 'request_id': 'req-996958f8-b264-4141-990d-a96a36daff98'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:22.989688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-53edc006-3812-4870-b2e8-65bcd859e45d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '390b054d9e7f4d6d9ed51656aff5d133'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:22.994169 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-996958f8-b264-4141-990d-a96a36daff98 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '82b48c489465430a836de26d4aa39440', 'request_id': 'req-996958f8-b264-4141-990d-a96a36daff98'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:23.064612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-53edc006-3812-4870-b2e8-65bcd859e45d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '390b054d9e7f4d6d9ed51656aff5d133'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:23.083516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1351/2710] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:54:22 2022] DELETE /identity/v3/users/390b054d9e7f4d6d9ed51656aff5d133 => generated 0 bytes in 413 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:23.155191 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-996958f8-b264-4141-990d-a96a36daff98 admin admin] The token cache is being invalidate because project 82b48c489465430a836de26d4aa39440 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:54:23.157108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-996958f8-b264-4141-990d-a96a36daff98 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:23.157108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-996958f8-b264-4141-990d-a96a36daff98 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:23.157679 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1360/2711] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:54:22 2022] DELETE /identity/v3/projects/82b48c489465430a836de26d4aa39440 => generated 0 bytes in 497 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:23.366819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53edc006-3812-4870-b2e8-65bcd859e45d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:23.368273 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-53edc006-3812-4870-b2e8-65bcd859e45d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:23.412913 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=18,keystone:SELECT=80,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:23.462017 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53edc006-3812-4870-b2e8-65bcd859e45d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:23.467026 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c7d8bd-36b7-4909-90fc-92e22c203e1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:23.467672 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c7d8bd-36b7-4909-90fc-92e22c203e1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:23.467954 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c7d8bd-36b7-4909-90fc-92e22c203e1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:23.468230 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c7d8bd-36b7-4909-90fc-92e22c203e1f admin admin] PATH_INFO: `/v3/projects/d64c50fedbf0431f84ad9ac73c3c5506` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:23.469449 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c7d8bd-36b7-4909-90fc-92e22c203e1f admin admin] RBAC: Authorizing `identity:delete_project(project_id=d64c50fedbf0431f84ad9ac73c3c5506)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:23.471962 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c7d8bd-36b7-4909-90fc-92e22c203e1f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:23.577433 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-97c7d8bd-36b7-4909-90fc-92e22c203e1f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd64c50fedbf0431f84ad9ac73c3c5506', 'request_id': 'req-97c7d8bd-36b7-4909-90fc-92e22c203e1f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:23.582898 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-97c7d8bd-36b7-4909-90fc-92e22c203e1f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd64c50fedbf0431f84ad9ac73c3c5506', 'request_id': 'req-97c7d8bd-36b7-4909-90fc-92e22c203e1f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:23.589102 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-97c7d8bd-36b7-4909-90fc-92e22c203e1f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd64c50fedbf0431f84ad9ac73c3c5506', 'request_id': 'req-97c7d8bd-36b7-4909-90fc-92e22c203e1f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:23.707780 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-97c7d8bd-36b7-4909-90fc-92e22c203e1f admin admin] The token cache is being invalidate because project d64c50fedbf0431f84ad9ac73c3c5506 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:54:23.708067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-97c7d8bd-36b7-4909-90fc-92e22c203e1f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:23.708703 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-97c7d8bd-36b7-4909-90fc-92e22c203e1f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:23.710212 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1352/2712] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:54:23 2022] DELETE /identity/v3/projects/d64c50fedbf0431f84ad9ac73c3c5506 => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:23.714366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996958f8-b264-4141-990d-a96a36daff98 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:23.716103 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-996958f8-b264-4141-990d-a96a36daff98 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:23.781807 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996958f8-b264-4141-990d-a96a36daff98 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:23.800011 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:23.837556 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c039e9dd-a488-4816-bdba-0b0112e690dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:23.838187 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c039e9dd-a488-4816-bdba-0b0112e690dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:23.838468 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c039e9dd-a488-4816-bdba-0b0112e690dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:23.838774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c039e9dd-a488-4816-bdba-0b0112e690dc admin admin] PATH_INFO: `/v3/users/89c82a12dca54a3e96ff6237120c5c4a` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:23.839887 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c039e9dd-a488-4816-bdba-0b0112e690dc admin admin] RBAC: Authorizing `identity:delete_user(user_id=89c82a12dca54a3e96ff6237120c5c4a)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:23.841877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c039e9dd-a488-4816-bdba-0b0112e690dc admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:24.013500 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-c039e9dd-a488-4816-bdba-0b0112e690dc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89c82a12dca54a3e96ff6237120c5c4a'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:24.026097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-c039e9dd-a488-4816-bdba-0b0112e690dc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89c82a12dca54a3e96ff6237120c5c4a'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:24.026800 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-c039e9dd-a488-4816-bdba-0b0112e690dc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89c82a12dca54a3e96ff6237120c5c4a'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:24.097215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-c039e9dd-a488-4816-bdba-0b0112e690dc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89c82a12dca54a3e96ff6237120c5c4a'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:24.104429 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-c039e9dd-a488-4816-bdba-0b0112e690dc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89c82a12dca54a3e96ff6237120c5c4a'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:24.110378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1361/2713] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:54:23 2022] DELETE /identity/v3/users/89c82a12dca54a3e96ff6237120c5c4a => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:24.339269 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c7d8bd-36b7-4909-90fc-92e22c203e1f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:24.347699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-97c7d8bd-36b7-4909-90fc-92e22c203e1f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:24.354252 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c7d8bd-36b7-4909-90fc-92e22c203e1f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:24.411797 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a1abb3-b26d-48f8-bcfa-ec3ed4fb69b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:24.412473 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a1abb3-b26d-48f8-bcfa-ec3ed4fb69b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:24.412689 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a1abb3-b26d-48f8-bcfa-ec3ed4fb69b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:24.412964 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a1abb3-b26d-48f8-bcfa-ec3ed4fb69b1 admin admin] PATH_INFO: `/v3/projects/be70ba68d13c4b9bbac7f0d788641990` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:24.414072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a1abb3-b26d-48f8-bcfa-ec3ed4fb69b1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=be70ba68d13c4b9bbac7f0d788641990)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:24.416068 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a1abb3-b26d-48f8-bcfa-ec3ed4fb69b1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:24.528284 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:24.529656 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-56a1abb3-b26d-48f8-bcfa-ec3ed4fb69b1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'be70ba68d13c4b9bbac7f0d788641990', 'request_id': 'req-56a1abb3-b26d-48f8-bcfa-ec3ed4fb69b1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:24.537680 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-56a1abb3-b26d-48f8-bcfa-ec3ed4fb69b1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'be70ba68d13c4b9bbac7f0d788641990', 'request_id': 'req-56a1abb3-b26d-48f8-bcfa-ec3ed4fb69b1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:24.544678 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-56a1abb3-b26d-48f8-bcfa-ec3ed4fb69b1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'be70ba68d13c4b9bbac7f0d788641990', 'request_id': 'req-56a1abb3-b26d-48f8-bcfa-ec3ed4fb69b1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:24.643221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-56a1abb3-b26d-48f8-bcfa-ec3ed4fb69b1 admin admin] The token cache is being invalidate because project be70ba68d13c4b9bbac7f0d788641990 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:54:24.644169 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-56a1abb3-b26d-48f8-bcfa-ec3ed4fb69b1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:24.644904 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-56a1abb3-b26d-48f8-bcfa-ec3ed4fb69b1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:24.646989 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1353/2714] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:54:24 2022] DELETE /identity/v3/projects/be70ba68d13c4b9bbac7f0d788641990 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:24.651738 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82482743-8747-4e83-88ba-05e66c85963b None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:24.652029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82482743-8747-4e83-88ba-05e66c85963b None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:24.652357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82482743-8747-4e83-88ba-05e66c85963b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:24.673322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-82482743-8747-4e83-88ba-05e66c85963b None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:54:24.677072 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-82482743-8747-4e83-88ba-05e66c85963b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:24.745075 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1362/2715] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:54:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:54:24.753337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec72fde-a90f-4ae2-a228-a8afb2054748 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:24.753886 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec72fde-a90f-4ae2-a228-a8afb2054748 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:24.754161 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec72fde-a90f-4ae2-a228-a8afb2054748 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:24.773629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9ec72fde-a90f-4ae2-a228-a8afb2054748 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:54:24.774864 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9ec72fde-a90f-4ae2-a228-a8afb2054748 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:24.782717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1354/2716] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:54:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:54:24.786939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82482743-8747-4e83-88ba-05e66c85963b None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:24.796576 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82482743-8747-4e83-88ba-05e66c85963b None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:24.801683 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a36a7c9-f39b-4a58-abaa-22e87117f58a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:24.802272 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a36a7c9-f39b-4a58-abaa-22e87117f58a admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:24.802553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a36a7c9-f39b-4a58-abaa-22e87117f58a admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:24.802855 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a36a7c9-f39b-4a58-abaa-22e87117f58a admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:24.803492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a36a7c9-f39b-4a58-abaa-22e87117f58a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:54:24.803792 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a36a7c9-f39b-4a58-abaa-22e87117f58a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:24.805378 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a36a7c9-f39b-4a58-abaa-22e87117f58a admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:24.817415 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1363/2717] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:54:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:54:24.821605 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec72fde-a90f-4ae2-a228-a8afb2054748 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:24.830828 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec72fde-a90f-4ae2-a228-a8afb2054748 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:24.835468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6731e692-d222-4cb9-a709-954a9dda1f92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:24.836057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6731e692-d222-4cb9-a709-954a9dda1f92 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:24.836336 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6731e692-d222-4cb9-a709-954a9dda1f92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:24.836610 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6731e692-d222-4cb9-a709-954a9dda1f92 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:24.837230 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6731e692-d222-4cb9-a709-954a9dda1f92 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:24.839196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6731e692-d222-4cb9-a709-954a9dda1f92 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:24.950538 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1355/2718] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:54:24 2022] POST /identity/v3/projects => generated 383 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:54:24.954488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a36a7c9-f39b-4a58-abaa-22e87117f58a admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:24.964543 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a36a7c9-f39b-4a58-abaa-22e87117f58a admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:24.990848 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:25.020661 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a742b41-ed6a-400e-b3dd-73641585f1d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:25.021302 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a742b41-ed6a-400e-b3dd-73641585f1d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:25.021579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a742b41-ed6a-400e-b3dd-73641585f1d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:25.021849 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a742b41-ed6a-400e-b3dd-73641585f1d9 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:25.022476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a742b41-ed6a-400e-b3dd-73641585f1d9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:25.024448 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a742b41-ed6a-400e-b3dd-73641585f1d9 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:25.104178 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1364/2719] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:54:24 2022] POST /identity/v3/users => generated 292 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:54:25.108271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6731e692-d222-4cb9-a709-954a9dda1f92 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:25.119323 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6731e692-d222-4cb9-a709-954a9dda1f92 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:25.123980 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c50870d-d540-41a6-bcd0-8ec9dadde9b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:25.124573 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c50870d-d540-41a6-bcd0-8ec9dadde9b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:25.124858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c50870d-d540-41a6-bcd0-8ec9dadde9b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:25.125136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c50870d-d540-41a6-bcd0-8ec9dadde9b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:25.125815 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c50870d-d540-41a6-bcd0-8ec9dadde9b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:25.127404 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c50870d-d540-41a6-bcd0-8ec9dadde9b2 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:25.137477 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1356/2720] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:54:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:54:25.141314 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a742b41-ed6a-400e-b3dd-73641585f1d9 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:25.151367 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a742b41-ed6a-400e-b3dd-73641585f1d9 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:25.156028 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22956edb-d295-4124-a514-14f9ce9ee287 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:25.156606 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22956edb-d295-4124-a514-14f9ce9ee287 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:25.156883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22956edb-d295-4124-a514-14f9ce9ee287 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:25.157151 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22956edb-d295-4124-a514-14f9ce9ee287 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:25.157774 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22956edb-d295-4124-a514-14f9ce9ee287 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:25.159934 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22956edb-d295-4124-a514-14f9ce9ee287 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:25.170001 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1365/2721] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:54:25 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:54:25.173600 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c50870d-d540-41a6-bcd0-8ec9dadde9b2 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:25.184332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c50870d-d540-41a6-bcd0-8ec9dadde9b2 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:25.189112 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461ed9c0-2381-4906-b7c2-23c54a26c547 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:25.189766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461ed9c0-2381-4906-b7c2-23c54a26c547 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:25.190043 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461ed9c0-2381-4906-b7c2-23c54a26c547 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:25.190315 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461ed9c0-2381-4906-b7c2-23c54a26c547 admin admin] PATH_INFO: `/v3/projects/2d30e521c04f4e85bd8d25c794704372/users/53f831616ca148648a7c9d5b10904fdb/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:25.204988 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461ed9c0-2381-4906-b7c2-23c54a26c547 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d30e521c04f4e85bd8d25c794704372, user_id=53f831616ca148648a7c9d5b10904fdb, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:25.207656 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461ed9c0-2381-4906-b7c2-23c54a26c547 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:25.287629 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1357/2722] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:54:25 2022] PUT /identity/v3/projects/2d30e521c04f4e85bd8d25c794704372/users/53f831616ca148648a7c9d5b10904fdb/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:25.353988 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9e89a0-4214-421f-a5b5-1e7406fdd99b None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:25.354270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9e89a0-4214-421f-a5b5-1e7406fdd99b None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:25.354543 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9e89a0-4214-421f-a5b5-1e7406fdd99b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:25.383404 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-1e9e89a0-4214-421f-a5b5-1e7406fdd99b None None] MFA Rules not processed for user `53f831616ca148648a7c9d5b10904fdb`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:54:25.384649 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-1e9e89a0-4214-421f-a5b5-1e7406fdd99b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:25.447634 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1366/2723] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:54:25 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:54:25.452263 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461ed9c0-2381-4906-b7c2-23c54a26c547 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:25.514207 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461ed9c0-2381-4906-b7c2-23c54a26c547 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:25.519421 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d985946-c04e-473d-8577-3b9bfda8eee8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:25.520014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d985946-c04e-473d-8577-3b9bfda8eee8 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:25.520292 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d985946-c04e-473d-8577-3b9bfda8eee8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:25.520561 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d985946-c04e-473d-8577-3b9bfda8eee8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:25.521188 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d985946-c04e-473d-8577-3b9bfda8eee8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:25.523117 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d985946-c04e-473d-8577-3b9bfda8eee8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:25.619790 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1358/2724] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:54:25 2022] POST /identity/v3/projects => generated 385 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:54:25.623987 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9e89a0-4214-421f-a5b5-1e7406fdd99b None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:25.635460 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9e89a0-4214-421f-a5b5-1e7406fdd99b None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:25.690942 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e842488a-266b-4f98-a4cb-420b4e75b054 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:25.691546 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e842488a-266b-4f98-a4cb-420b4e75b054 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:25.691827 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e842488a-266b-4f98-a4cb-420b4e75b054 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:25.692100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e842488a-266b-4f98-a4cb-420b4e75b054 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:25.692716 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e842488a-266b-4f98-a4cb-420b4e75b054 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:25.694672 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e842488a-266b-4f98-a4cb-420b4e75b054 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:25.807019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1367/2725] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:54:25 2022] POST /identity/v3/users => generated 299 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:54:25.810872 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d985946-c04e-473d-8577-3b9bfda8eee8 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:25.821684 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d985946-c04e-473d-8577-3b9bfda8eee8 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:25.826510 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de07e10-8fc1-4e29-808a-b14e6874b430 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:25.826965 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de07e10-8fc1-4e29-808a-b14e6874b430 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:25.827245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de07e10-8fc1-4e29-808a-b14e6874b430 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:25.827516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de07e10-8fc1-4e29-808a-b14e6874b430 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:25.828149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de07e10-8fc1-4e29-808a-b14e6874b430 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:25.829742 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de07e10-8fc1-4e29-808a-b14e6874b430 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:25.840694 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1359/2726] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:54:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:54:25.844666 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e842488a-266b-4f98-a4cb-420b4e75b054 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:25.854802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e842488a-266b-4f98-a4cb-420b4e75b054 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:25.859441 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f0b2e2-1536-454f-b2ea-04ad910149df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:25.860087 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f0b2e2-1536-454f-b2ea-04ad910149df admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:25.860366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f0b2e2-1536-454f-b2ea-04ad910149df admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:25.860638 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f0b2e2-1536-454f-b2ea-04ad910149df admin admin] PATH_INFO: `/v3/projects/b9f56079af3f4222a5673d4cd1eab194/users/cc01392798144880a4896bcbacddd91e/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:25.875284 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f0b2e2-1536-454f-b2ea-04ad910149df admin admin] RBAC: Authorizing `identity:create_grant(project_id=b9f56079af3f4222a5673d4cd1eab194, user_id=cc01392798144880a4896bcbacddd91e, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:25.878840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f0b2e2-1536-454f-b2ea-04ad910149df admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:25.972603 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1368/2727] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:54:25 2022] PUT /identity/v3/projects/b9f56079af3f4222a5673d4cd1eab194/users/cc01392798144880a4896bcbacddd91e/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:54:25.976747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de07e10-8fc1-4e29-808a-b14e6874b430 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:25.987352 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de07e10-8fc1-4e29-808a-b14e6874b430 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:26.030277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=93,keystone:DELETE=3,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:26.042609 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26dc675-515b-4038-9ef2-4754ff67cce9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:26.043213 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26dc675-515b-4038-9ef2-4754ff67cce9 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:26.043492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26dc675-515b-4038-9ef2-4754ff67cce9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:26.043761 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26dc675-515b-4038-9ef2-4754ff67cce9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:26.044390 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26dc675-515b-4038-9ef2-4754ff67cce9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:26.046028 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26dc675-515b-4038-9ef2-4754ff67cce9 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:26.055947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1360/2728] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:54:25 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:54:26.060270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f0b2e2-1536-454f-b2ea-04ad910149df admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:26.066266 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:26.074823 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f0b2e2-1536-454f-b2ea-04ad910149df admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:26.076192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54addae-7507-4f1a-a135-7b3169286f84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:26.076839 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54addae-7507-4f1a-a135-7b3169286f84 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:26.077117 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54addae-7507-4f1a-a135-7b3169286f84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:26.077389 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54addae-7507-4f1a-a135-7b3169286f84 admin admin] PATH_INFO: `/v3/projects/b9f56079af3f4222a5673d4cd1eab194/users/cc01392798144880a4896bcbacddd91e/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:26.079297 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54addae-7507-4f1a-a135-7b3169286f84 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b9f56079af3f4222a5673d4cd1eab194, user_id=cc01392798144880a4896bcbacddd91e, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:26.082008 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54addae-7507-4f1a-a135-7b3169286f84 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:26.092102 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1369/2729] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:54:26 2022] PUT /identity/v3/projects/b9f56079af3f4222a5673d4cd1eab194/users/cc01392798144880a4896bcbacddd91e/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:54:26.101998 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24ce840-0288-4d60-8bbf-9fdf25e1c7e2 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:26.102284 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24ce840-0288-4d60-8bbf-9fdf25e1c7e2 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:26.102571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24ce840-0288-4d60-8bbf-9fdf25e1c7e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:26.131346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c24ce840-0288-4d60-8bbf-9fdf25e1c7e2 None None] MFA Rules not processed for user `cc01392798144880a4896bcbacddd91e`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:54:26.132579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c24ce840-0288-4d60-8bbf-9fdf25e1c7e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:26.199235 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1361/2730] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:54:26 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:54:26.209651 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54addae-7507-4f1a-a135-7b3169286f84 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:26.211324 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-f54addae-7507-4f1a-a135-7b3169286f84 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:26.268316 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54addae-7507-4f1a-a135-7b3169286f84 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:26.271921 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f6544b-5550-40af-8ef7-3c1dd3e8add3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:26.272509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f6544b-5550-40af-8ef7-3c1dd3e8add3 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:26.272787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f6544b-5550-40af-8ef7-3c1dd3e8add3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:26.273056 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f6544b-5550-40af-8ef7-3c1dd3e8add3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:26.327419 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f6544b-5550-40af-8ef7-3c1dd3e8add3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:26.328781 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f6544b-5550-40af-8ef7-3c1dd3e8add3 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:26.337391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1370/2731] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:54:26 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:54:31.398225 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24ce840-0288-4d60-8bbf-9fdf25e1c7e2 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:31.400128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c24ce840-0288-4d60-8bbf-9fdf25e1c7e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:31.412350 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24ce840-0288-4d60-8bbf-9fdf25e1c7e2 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:31.415922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33229464-6887-4999-8619-2562722e4791 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:31.416518 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33229464-6887-4999-8619-2562722e4791 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:31.416798 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33229464-6887-4999-8619-2562722e4791 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:31.417072 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33229464-6887-4999-8619-2562722e4791 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:31.427137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33229464-6887-4999-8619-2562722e4791 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:31.428571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33229464-6887-4999-8619-2562722e4791 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:31.438135 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1362/2732] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:54:31 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:54:36.333471 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=52,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:37.373029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f6544b-5550-40af-8ef7-3c1dd3e8add3 service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:37.433689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f6544b-5550-40af-8ef7-3c1dd3e8add3 service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:37.438151 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbb8305-3b1a-48c5-9e5a-33c71c40b213 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:37.438805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbb8305-3b1a-48c5-9e5a-33c71c40b213 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:37.439085 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbb8305-3b1a-48c5-9e5a-33c71c40b213 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:37.439354 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbb8305-3b1a-48c5-9e5a-33c71c40b213 admin admin] PATH_INFO: `/v3/users/53f831616ca148648a7c9d5b10904fdb` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:37.440446 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbb8305-3b1a-48c5-9e5a-33c71c40b213 admin admin] RBAC: Authorizing `identity:delete_user(user_id=53f831616ca148648a7c9d5b10904fdb)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:37.442366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbb8305-3b1a-48c5-9e5a-33c71c40b213 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:37.663997 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0fbb8305-3b1a-48c5-9e5a-33c71c40b213 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '53f831616ca148648a7c9d5b10904fdb'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:37.675633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0fbb8305-3b1a-48c5-9e5a-33c71c40b213 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '53f831616ca148648a7c9d5b10904fdb'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:37.676284 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0fbb8305-3b1a-48c5-9e5a-33c71c40b213 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '53f831616ca148648a7c9d5b10904fdb'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:37.750107 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0fbb8305-3b1a-48c5-9e5a-33c71c40b213 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '53f831616ca148648a7c9d5b10904fdb'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:37.758428 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0fbb8305-3b1a-48c5-9e5a-33c71c40b213 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '53f831616ca148648a7c9d5b10904fdb'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:37.764384 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1371/2733] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:54:37 2022] DELETE /identity/v3/users/53f831616ca148648a7c9d5b10904fdb => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:37.768811 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca2977f-624d-4124-b44a-7180f546225a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:37.769885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca2977f-624d-4124-b44a-7180f546225a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:37.769993 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca2977f-624d-4124-b44a-7180f546225a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:37.788958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7ca2977f-624d-4124-b44a-7180f546225a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:54:37.790154 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7ca2977f-624d-4124-b44a-7180f546225a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:37.847448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1363/2734] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:54:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:54:37.857029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbb8305-3b1a-48c5-9e5a-33c71c40b213 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:37.858425 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0fbb8305-3b1a-48c5-9e5a-33c71c40b213 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:37.916098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbb8305-3b1a-48c5-9e5a-33c71c40b213 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:37.919633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18edf380-8f78-472a-a8f0-163c1bb93a0e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:37.920235 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18edf380-8f78-472a-a8f0-163c1bb93a0e service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:37.920517 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18edf380-8f78-472a-a8f0-163c1bb93a0e service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:37.920792 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18edf380-8f78-472a-a8f0-163c1bb93a0e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:37.929671 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18edf380-8f78-472a-a8f0-163c1bb93a0e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:37.931152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18edf380-8f78-472a-a8f0-163c1bb93a0e service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:37.940577 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1372/2735] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:54:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:54:38.321725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca2977f-624d-4124-b44a-7180f546225a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:38.326040 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7ca2977f-624d-4124-b44a-7180f546225a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:38.336957 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca2977f-624d-4124-b44a-7180f546225a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:38.341898 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caed758b-bf64-4018-b688-187a883f021f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:38.342541 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caed758b-bf64-4018-b688-187a883f021f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:38.342860 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caed758b-bf64-4018-b688-187a883f021f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:38.343139 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caed758b-bf64-4018-b688-187a883f021f admin admin] PATH_INFO: `/v3/projects/2d30e521c04f4e85bd8d25c794704372` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:38.344351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caed758b-bf64-4018-b688-187a883f021f admin admin] RBAC: Authorizing `identity:delete_project(project_id=2d30e521c04f4e85bd8d25c794704372)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:38.346467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caed758b-bf64-4018-b688-187a883f021f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:38.467807 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-caed758b-bf64-4018-b688-187a883f021f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2d30e521c04f4e85bd8d25c794704372', 'request_id': 'req-caed758b-bf64-4018-b688-187a883f021f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:38.468918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:38.474167 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-caed758b-bf64-4018-b688-187a883f021f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2d30e521c04f4e85bd8d25c794704372', 'request_id': 'req-caed758b-bf64-4018-b688-187a883f021f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:38.480195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-caed758b-bf64-4018-b688-187a883f021f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2d30e521c04f4e85bd8d25c794704372', 'request_id': 'req-caed758b-bf64-4018-b688-187a883f021f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:38.541287 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-caed758b-bf64-4018-b688-187a883f021f admin admin] The token cache is being invalidate because project 2d30e521c04f4e85bd8d25c794704372 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:54:38.541571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-caed758b-bf64-4018-b688-187a883f021f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:38.542199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-caed758b-bf64-4018-b688-187a883f021f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:38.543682 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1364/2736] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:54:38 2022] DELETE /identity/v3/projects/2d30e521c04f4e85bd8d25c794704372 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:38.547730 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18edf380-8f78-472a-a8f0-163c1bb93a0e service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:38.557785 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18edf380-8f78-472a-a8f0-163c1bb93a0e service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:38.575505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:38.613703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ab2b66-7b3a-4b0b-9898-2d9ae285f6d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:38.614361 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ab2b66-7b3a-4b0b-9898-2d9ae285f6d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:38.614691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ab2b66-7b3a-4b0b-9898-2d9ae285f6d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:38.614973 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ab2b66-7b3a-4b0b-9898-2d9ae285f6d8 admin admin] PATH_INFO: `/v3/users/cc01392798144880a4896bcbacddd91e` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:38.616143 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ab2b66-7b3a-4b0b-9898-2d9ae285f6d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cc01392798144880a4896bcbacddd91e)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:38.618101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ab2b66-7b3a-4b0b-9898-2d9ae285f6d8 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:38.806520 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-19ab2b66-7b3a-4b0b-9898-2d9ae285f6d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cc01392798144880a4896bcbacddd91e'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:38.818758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-19ab2b66-7b3a-4b0b-9898-2d9ae285f6d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cc01392798144880a4896bcbacddd91e'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:38.819421 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-19ab2b66-7b3a-4b0b-9898-2d9ae285f6d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cc01392798144880a4896bcbacddd91e'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:38.869712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-19ab2b66-7b3a-4b0b-9898-2d9ae285f6d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cc01392798144880a4896bcbacddd91e'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:38.876664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-19ab2b66-7b3a-4b0b-9898-2d9ae285f6d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cc01392798144880a4896bcbacddd91e'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:38.882740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1373/2737] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:54:38 2022] DELETE /identity/v3/users/cc01392798144880a4896bcbacddd91e => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:39.270975 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caed758b-bf64-4018-b688-187a883f021f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:39.272586 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-caed758b-bf64-4018-b688-187a883f021f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:39.286981 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caed758b-bf64-4018-b688-187a883f021f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:39.348084 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03eeb54-ab1b-49b2-8213-8877a5dc3019 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:39.348639 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03eeb54-ab1b-49b2-8213-8877a5dc3019 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:39.348916 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03eeb54-ab1b-49b2-8213-8877a5dc3019 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:39.349186 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03eeb54-ab1b-49b2-8213-8877a5dc3019 admin admin] PATH_INFO: `/v3/projects/b9f56079af3f4222a5673d4cd1eab194` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:39.350286 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03eeb54-ab1b-49b2-8213-8877a5dc3019 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b9f56079af3f4222a5673d4cd1eab194)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:39.352221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03eeb54-ab1b-49b2-8213-8877a5dc3019 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:39.471858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a03eeb54-ab1b-49b2-8213-8877a5dc3019 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b9f56079af3f4222a5673d4cd1eab194', 'request_id': 'req-a03eeb54-ab1b-49b2-8213-8877a5dc3019'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:39.477587 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a03eeb54-ab1b-49b2-8213-8877a5dc3019 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b9f56079af3f4222a5673d4cd1eab194', 'request_id': 'req-a03eeb54-ab1b-49b2-8213-8877a5dc3019'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:39.483815 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a03eeb54-ab1b-49b2-8213-8877a5dc3019 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b9f56079af3f4222a5673d4cd1eab194', 'request_id': 'req-a03eeb54-ab1b-49b2-8213-8877a5dc3019'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:39.572932 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a03eeb54-ab1b-49b2-8213-8877a5dc3019 admin admin] The token cache is being invalidate because project b9f56079af3f4222a5673d4cd1eab194 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:54:39.573219 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a03eeb54-ab1b-49b2-8213-8877a5dc3019 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:39.573956 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a03eeb54-ab1b-49b2-8213-8877a5dc3019 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:39.575635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1365/2738] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:54:39 2022] DELETE /identity/v3/projects/b9f56079af3f4222a5673d4cd1eab194 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:39.581911 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b3be87-a48e-482e-8869-84bfba8287a4 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:39.582454 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b3be87-a48e-482e-8869-84bfba8287a4 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:39.582454 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b3be87-a48e-482e-8869-84bfba8287a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:39.602582 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-29b3be87-a48e-482e-8869-84bfba8287a4 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:54:39.603877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-29b3be87-a48e-482e-8869-84bfba8287a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:39.663549 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1374/2739] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:54:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:54:39.671939 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c283f9f6-09be-4250-90ea-84d6e963dfb1 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:39.672226 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c283f9f6-09be-4250-90ea-84d6e963dfb1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:39.672509 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c283f9f6-09be-4250-90ea-84d6e963dfb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:39.692933 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c283f9f6-09be-4250-90ea-84d6e963dfb1 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:54:39.694128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c283f9f6-09be-4250-90ea-84d6e963dfb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:39.701926 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1366/2740] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:54:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:54:39.705933 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b3be87-a48e-482e-8869-84bfba8287a4 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:39.715725 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b3be87-a48e-482e-8869-84bfba8287a4 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:39.720409 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdba166-13bc-4f9a-9068-019cd732f835 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:39.721000 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdba166-13bc-4f9a-9068-019cd732f835 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:39.721279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdba166-13bc-4f9a-9068-019cd732f835 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:39.721557 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdba166-13bc-4f9a-9068-019cd732f835 admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:39.722190 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcdba166-13bc-4f9a-9068-019cd732f835 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:54:39.722492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcdba166-13bc-4f9a-9068-019cd732f835 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:39.724132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcdba166-13bc-4f9a-9068-019cd732f835 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:39.735583 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1375/2741] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:54:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:54:39.740311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c283f9f6-09be-4250-90ea-84d6e963dfb1 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:39.749405 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c283f9f6-09be-4250-90ea-84d6e963dfb1 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:39.754859 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d566e9f8-a02d-4f1f-9fb4-876f0a3a83c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:39.755448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d566e9f8-a02d-4f1f-9fb4-876f0a3a83c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:39.755725 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d566e9f8-a02d-4f1f-9fb4-876f0a3a83c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:39.755995 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d566e9f8-a02d-4f1f-9fb4-876f0a3a83c8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:39.756630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d566e9f8-a02d-4f1f-9fb4-876f0a3a83c8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:39.758500 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d566e9f8-a02d-4f1f-9fb4-876f0a3a83c8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:39.843001 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1367/2742] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:54:39 2022] POST /identity/v3/projects => generated 371 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:54:39.847536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdba166-13bc-4f9a-9068-019cd732f835 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:39.857604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdba166-13bc-4f9a-9068-019cd732f835 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:39.883813 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:39.913868 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de118d19-88be-4fe6-bd84-0fb3a20df66a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:39.914515 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de118d19-88be-4fe6-bd84-0fb3a20df66a admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:39.914828 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de118d19-88be-4fe6-bd84-0fb3a20df66a admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:39.915104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de118d19-88be-4fe6-bd84-0fb3a20df66a admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:39.915750 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de118d19-88be-4fe6-bd84-0fb3a20df66a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:39.917673 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de118d19-88be-4fe6-bd84-0fb3a20df66a admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:39.999215 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1376/2743] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:54:39 2022] POST /identity/v3/users => generated 286 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:54:40.003296 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d566e9f8-a02d-4f1f-9fb4-876f0a3a83c8 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:40.013779 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d566e9f8-a02d-4f1f-9fb4-876f0a3a83c8 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:40.018397 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd819e69-7f9e-478d-8b8d-09dcafe431d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:40.019062 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd819e69-7f9e-478d-8b8d-09dcafe431d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:40.019310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd819e69-7f9e-478d-8b8d-09dcafe431d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:40.019579 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd819e69-7f9e-478d-8b8d-09dcafe431d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:40.020214 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd819e69-7f9e-478d-8b8d-09dcafe431d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:40.021844 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd819e69-7f9e-478d-8b8d-09dcafe431d9 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:40.031634 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1368/2744] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:54:40 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:54:40.035973 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de118d19-88be-4fe6-bd84-0fb3a20df66a admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:40.045932 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de118d19-88be-4fe6-bd84-0fb3a20df66a admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:40.051255 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d9fe74-4fc8-463c-9275-c5625e706a59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:40.051842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d9fe74-4fc8-463c-9275-c5625e706a59 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:40.052128 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d9fe74-4fc8-463c-9275-c5625e706a59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:40.052404 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d9fe74-4fc8-463c-9275-c5625e706a59 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:40.053040 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d9fe74-4fc8-463c-9275-c5625e706a59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:40.054675 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d9fe74-4fc8-463c-9275-c5625e706a59 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:40.065221 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1377/2745] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:54:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:54:40.069444 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd819e69-7f9e-478d-8b8d-09dcafe431d9 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:40.079199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd819e69-7f9e-478d-8b8d-09dcafe431d9 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:40.083747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d33f9fe-b185-4347-8342-1d0411d63f65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:40.084381 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d33f9fe-b185-4347-8342-1d0411d63f65 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:40.084660 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d33f9fe-b185-4347-8342-1d0411d63f65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:40.084929 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d33f9fe-b185-4347-8342-1d0411d63f65 admin admin] PATH_INFO: `/v3/projects/609696f6bc224cb9ad0532f2ff62b926/users/9908486062b94cabac31c848aaf5b4cf/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:40.099499 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d33f9fe-b185-4347-8342-1d0411d63f65 admin admin] RBAC: Authorizing `identity:create_grant(project_id=609696f6bc224cb9ad0532f2ff62b926, user_id=9908486062b94cabac31c848aaf5b4cf, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:40.102134 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d33f9fe-b185-4347-8342-1d0411d63f65 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:40.188157 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1369/2746] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:54:40 2022] PUT /identity/v3/projects/609696f6bc224cb9ad0532f2ff62b926/users/9908486062b94cabac31c848aaf5b4cf/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:40.200468 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f00938-9c7f-42b5-9b66-4b63abf04c18 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:40.200759 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f00938-9c7f-42b5-9b66-4b63abf04c18 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:40.201035 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f00938-9c7f-42b5-9b66-4b63abf04c18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:40.229299 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-73f00938-9c7f-42b5-9b66-4b63abf04c18 None None] MFA Rules not processed for user `9908486062b94cabac31c848aaf5b4cf`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:54:40.230498 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-73f00938-9c7f-42b5-9b66-4b63abf04c18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:40.292298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1378/2747] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:54:40 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:54:40.303632 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d33f9fe-b185-4347-8342-1d0411d63f65 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:40.306272 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9d33f9fe-b185-4347-8342-1d0411d63f65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:40.312832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=10,keystone:SELECT=86,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:40.363106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d33f9fe-b185-4347-8342-1d0411d63f65 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:40.366787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45a6d32-79ca-4e53-9235-90b02fff8da2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:40.367387 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45a6d32-79ca-4e53-9235-90b02fff8da2 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:40.367669 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45a6d32-79ca-4e53-9235-90b02fff8da2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:40.367944 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45a6d32-79ca-4e53-9235-90b02fff8da2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:40.376564 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45a6d32-79ca-4e53-9235-90b02fff8da2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:40.378023 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45a6d32-79ca-4e53-9235-90b02fff8da2 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:40.387168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1370/2748] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:54:40 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:54:40.408991 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f00938-9c7f-42b5-9b66-4b63abf04c18 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:40.469028 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f00938-9c7f-42b5-9b66-4b63abf04c18 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:40.473988 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e02db05-0191-4de0-a738-3617526d435d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:40.474818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e02db05-0191-4de0-a738-3617526d435d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:40.475096 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e02db05-0191-4de0-a738-3617526d435d admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:40.475368 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e02db05-0191-4de0-a738-3617526d435d admin admin] PATH_INFO: `/v3/users/9908486062b94cabac31c848aaf5b4cf` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:40.476497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e02db05-0191-4de0-a738-3617526d435d admin admin] RBAC: Authorizing `identity:delete_user(user_id=9908486062b94cabac31c848aaf5b4cf)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:40.478409 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e02db05-0191-4de0-a738-3617526d435d admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:40.660796 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:40.667175 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0e02db05-0191-4de0-a738-3617526d435d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9908486062b94cabac31c848aaf5b4cf'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:40.679384 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0e02db05-0191-4de0-a738-3617526d435d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9908486062b94cabac31c848aaf5b4cf'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:40.680234 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0e02db05-0191-4de0-a738-3617526d435d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9908486062b94cabac31c848aaf5b4cf'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:40.744266 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0e02db05-0191-4de0-a738-3617526d435d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9908486062b94cabac31c848aaf5b4cf'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:40.751883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-0e02db05-0191-4de0-a738-3617526d435d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9908486062b94cabac31c848aaf5b4cf'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:40.758652 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1379/2749] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:54:40 2022] DELETE /identity/v3/users/9908486062b94cabac31c848aaf5b4cf => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:40.763314 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fda5861-fb63-4b19-bbc4-ba00f509b903 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:40.763720 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fda5861-fb63-4b19-bbc4-ba00f509b903 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:40.764066 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fda5861-fb63-4b19-bbc4-ba00f509b903 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:40.786632 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-4fda5861-fb63-4b19-bbc4-ba00f509b903 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:54:40.787939 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4fda5861-fb63-4b19-bbc4-ba00f509b903 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:40.847921 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1371/2750] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:54:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:54:40.857576 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e02db05-0191-4de0-a738-3617526d435d admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:40.859116 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0e02db05-0191-4de0-a738-3617526d435d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:40.916601 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e02db05-0191-4de0-a738-3617526d435d admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:40.920240 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f91b517-4afe-4bff-b3a9-621a752bf3bb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:40.920836 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f91b517-4afe-4bff-b3a9-621a752bf3bb service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:40.921113 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f91b517-4afe-4bff-b3a9-621a752bf3bb service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:40.921385 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f91b517-4afe-4bff-b3a9-621a752bf3bb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:40.930639 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f91b517-4afe-4bff-b3a9-621a752bf3bb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:40.932929 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f91b517-4afe-4bff-b3a9-621a752bf3bb service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:40.942004 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1380/2751] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:54:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:54:41.671645 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fda5861-fb63-4b19-bbc4-ba00f509b903 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:41.673096 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4fda5861-fb63-4b19-bbc4-ba00f509b903 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:41.686697 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fda5861-fb63-4b19-bbc4-ba00f509b903 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:41.693048 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497a2db4-e229-4c09-ade9-8b81952e23b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:41.693699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497a2db4-e229-4c09-ade9-8b81952e23b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:41.693974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497a2db4-e229-4c09-ade9-8b81952e23b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:41.694249 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497a2db4-e229-4c09-ade9-8b81952e23b3 admin admin] PATH_INFO: `/v3/projects/609696f6bc224cb9ad0532f2ff62b926` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:41.696084 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497a2db4-e229-4c09-ade9-8b81952e23b3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=609696f6bc224cb9ad0532f2ff62b926)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:41.698821 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497a2db4-e229-4c09-ade9-8b81952e23b3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:41.801257 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-497a2db4-e229-4c09-ade9-8b81952e23b3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '609696f6bc224cb9ad0532f2ff62b926', 'request_id': 'req-497a2db4-e229-4c09-ade9-8b81952e23b3'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:41.806909 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-497a2db4-e229-4c09-ade9-8b81952e23b3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '609696f6bc224cb9ad0532f2ff62b926', 'request_id': 'req-497a2db4-e229-4c09-ade9-8b81952e23b3'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:41.813145 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-497a2db4-e229-4c09-ade9-8b81952e23b3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '609696f6bc224cb9ad0532f2ff62b926', 'request_id': 'req-497a2db4-e229-4c09-ade9-8b81952e23b3'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:41.880607 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-497a2db4-e229-4c09-ade9-8b81952e23b3 admin admin] The token cache is being invalidate because project 609696f6bc224cb9ad0532f2ff62b926 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:54:41.880917 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-497a2db4-e229-4c09-ade9-8b81952e23b3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:41.881574 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-497a2db4-e229-4c09-ade9-8b81952e23b3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:41.883359 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1372/2752] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:54:41 2022] DELETE /identity/v3/projects/609696f6bc224cb9ad0532f2ff62b926 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:41.888252 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255a0cbc-8d2c-427c-aa32-a078dc92396e None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:41.888799 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255a0cbc-8d2c-427c-aa32-a078dc92396e None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:41.889054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255a0cbc-8d2c-427c-aa32-a078dc92396e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:41.913619 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-255a0cbc-8d2c-427c-aa32-a078dc92396e None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:54:41.915125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-255a0cbc-8d2c-427c-aa32-a078dc92396e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:42.000648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1381/2753] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:54:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:54:42.008867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e37aff3-5890-4ecd-bc25-ca82b8e6486c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:42.010575 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e37aff3-5890-4ecd-bc25-ca82b8e6486c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:42.010575 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e37aff3-5890-4ecd-bc25-ca82b8e6486c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:42.030433 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9e37aff3-5890-4ecd-bc25-ca82b8e6486c None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:54:42.031684 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9e37aff3-5890-4ecd-bc25-ca82b8e6486c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:42.040677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1373/2754] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:54:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:54:42.044916 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255a0cbc-8d2c-427c-aa32-a078dc92396e None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:42.054357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255a0cbc-8d2c-427c-aa32-a078dc92396e None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:42.059013 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8500d2e-2a90-44a9-9578-a6b2e0e54d4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:42.059597 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8500d2e-2a90-44a9-9578-a6b2e0e54d4f admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:42.059875 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8500d2e-2a90-44a9-9578-a6b2e0e54d4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:42.060145 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8500d2e-2a90-44a9-9578-a6b2e0e54d4f admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:42.060777 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8500d2e-2a90-44a9-9578-a6b2e0e54d4f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:54:42.061072 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8500d2e-2a90-44a9-9578-a6b2e0e54d4f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:42.062689 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8500d2e-2a90-44a9-9578-a6b2e0e54d4f admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:42.074434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1382/2755] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:54:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:54:42.082843 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e37aff3-5890-4ecd-bc25-ca82b8e6486c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:42.092210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e37aff3-5890-4ecd-bc25-ca82b8e6486c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:42.096918 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c62928-4c3f-4ede-9913-e65713cd92f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:42.097524 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c62928-4c3f-4ede-9913-e65713cd92f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:42.097805 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c62928-4c3f-4ede-9913-e65713cd92f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:42.098077 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c62928-4c3f-4ede-9913-e65713cd92f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:42.098868 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c62928-4c3f-4ede-9913-e65713cd92f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:42.100743 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c62928-4c3f-4ede-9913-e65713cd92f3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:42.215503 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1374/2756] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:54:42 2022] POST /identity/v3/projects => generated 385 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:54:42.219710 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8500d2e-2a90-44a9-9578-a6b2e0e54d4f admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:42.229723 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8500d2e-2a90-44a9-9578-a6b2e0e54d4f admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:42.275853 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:DELETE=5,keystone:SELECT=94,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:42.288260 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbe92c7-d60f-4818-bd41-e32338140e13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:42.288873 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbe92c7-d60f-4818-bd41-e32338140e13 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:42.289153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbe92c7-d60f-4818-bd41-e32338140e13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:42.289422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbe92c7-d60f-4818-bd41-e32338140e13 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:42.290040 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fbe92c7-d60f-4818-bd41-e32338140e13 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:42.292192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fbe92c7-d60f-4818-bd41-e32338140e13 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:42.395280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1383/2757] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:54:42 2022] POST /identity/v3/users => generated 293 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:54:42.396968 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c62928-4c3f-4ede-9913-e65713cd92f3 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:42.412738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:42.423992 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c62928-4c3f-4ede-9913-e65713cd92f3 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:42.425766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029f16bb-9b40-4ae0-9d0a-b02c61f8c42a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:42.426464 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029f16bb-9b40-4ae0-9d0a-b02c61f8c42a admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:42.426841 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029f16bb-9b40-4ae0-9d0a-b02c61f8c42a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:42.427181 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029f16bb-9b40-4ae0-9d0a-b02c61f8c42a admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:42.427893 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029f16bb-9b40-4ae0-9d0a-b02c61f8c42a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:42.429602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029f16bb-9b40-4ae0-9d0a-b02c61f8c42a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:42.442621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1375/2758] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:54:42 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:54:42.446510 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbe92c7-d60f-4818-bd41-e32338140e13 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:42.462997 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbe92c7-d60f-4818-bd41-e32338140e13 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:42.471450 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a43d10-f463-4ff8-afd2-f9e044b1d857 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:42.472078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a43d10-f463-4ff8-afd2-f9e044b1d857 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:42.472322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a43d10-f463-4ff8-afd2-f9e044b1d857 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:42.472642 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a43d10-f463-4ff8-afd2-f9e044b1d857 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:42.473291 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a43d10-f463-4ff8-afd2-f9e044b1d857 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:42.474961 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a43d10-f463-4ff8-afd2-f9e044b1d857 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:42.496924 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1384/2759] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:54:42 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:54:42.498197 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029f16bb-9b40-4ae0-9d0a-b02c61f8c42a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:42.510631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029f16bb-9b40-4ae0-9d0a-b02c61f8c42a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:42.515746 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fee730-2bb7-4b3a-b882-c4c7c5676f9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:42.516509 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fee730-2bb7-4b3a-b882-c4c7c5676f9e admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:42.517363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fee730-2bb7-4b3a-b882-c4c7c5676f9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:42.517363 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fee730-2bb7-4b3a-b882-c4c7c5676f9e admin admin] PATH_INFO: `/v3/projects/9512c7b73e85419a86ba54d674de615a/users/35e97cc914e4413fa4af8bcf4a3d5fae/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:42.534731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fee730-2bb7-4b3a-b882-c4c7c5676f9e admin admin] RBAC: Authorizing `identity:create_grant(project_id=9512c7b73e85419a86ba54d674de615a, user_id=35e97cc914e4413fa4af8bcf4a3d5fae, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:42.538449 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fee730-2bb7-4b3a-b882-c4c7c5676f9e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:42.658937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1376/2760] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:54:42 2022] PUT /identity/v3/projects/9512c7b73e85419a86ba54d674de615a/users/35e97cc914e4413fa4af8bcf4a3d5fae/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:42.671157 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7e197f-bc5e-4e8c-9919-110e101c9318 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:42.671157 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7e197f-bc5e-4e8c-9919-110e101c9318 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:42.671486 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7e197f-bc5e-4e8c-9919-110e101c9318 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:42.699888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-cf7e197f-bc5e-4e8c-9919-110e101c9318 None None] MFA Rules not processed for user `35e97cc914e4413fa4af8bcf4a3d5fae`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:54:42.701153 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-cf7e197f-bc5e-4e8c-9919-110e101c9318 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:42.767304 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1385/2761] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:54:42 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:54:42.771906 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fee730-2bb7-4b3a-b882-c4c7c5676f9e admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:42.835475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fee730-2bb7-4b3a-b882-c4c7c5676f9e admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:42.840658 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e5f6f4-eddb-4e1e-aae1-163f44779d0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:42.841264 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e5f6f4-eddb-4e1e-aae1-163f44779d0c admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:42.841542 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e5f6f4-eddb-4e1e-aae1-163f44779d0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:42.841813 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e5f6f4-eddb-4e1e-aae1-163f44779d0c admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:42.842444 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e5f6f4-eddb-4e1e-aae1-163f44779d0c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:42.844504 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e5f6f4-eddb-4e1e-aae1-163f44779d0c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:43.022191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1377/2762] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:54:42 2022] POST /identity/v3/projects => generated 383 bytes in 251 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:54:43.027692 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7e197f-bc5e-4e8c-9919-110e101c9318 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:43.044842 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7e197f-bc5e-4e8c-9919-110e101c9318 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:43.113849 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86112974-d713-427d-b2cf-97028ebfe8a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:43.114448 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86112974-d713-427d-b2cf-97028ebfe8a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:43.114767 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86112974-d713-427d-b2cf-97028ebfe8a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:43.115039 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86112974-d713-427d-b2cf-97028ebfe8a1 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:43.115677 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86112974-d713-427d-b2cf-97028ebfe8a1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:43.117665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86112974-d713-427d-b2cf-97028ebfe8a1 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:43.256054 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1386/2763] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:54:43 2022] POST /identity/v3/users => generated 292 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:54:43.259431 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e5f6f4-eddb-4e1e-aae1-163f44779d0c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:43.262319 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86112974-d713-427d-b2cf-97028ebfe8a1 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:43.263986 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-86112974-d713-427d-b2cf-97028ebfe8a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:43.270678 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e5f6f4-eddb-4e1e-aae1-163f44779d0c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:43.275446 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b98145-5e31-47ec-af99-fd2115304035 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:43.276063 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b98145-5e31-47ec-af99-fd2115304035 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:43.276344 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b98145-5e31-47ec-af99-fd2115304035 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:43.276617 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b98145-5e31-47ec-af99-fd2115304035 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:43.277268 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b98145-5e31-47ec-af99-fd2115304035 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:43.279412 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b98145-5e31-47ec-af99-fd2115304035 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:43.291815 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1378/2764] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:54:43 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:54:43.295793 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b98145-5e31-47ec-af99-fd2115304035 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:43.307228 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b98145-5e31-47ec-af99-fd2115304035 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:43.312221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91362c3-b88b-4543-95e9-a5bc9f53634f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:43.313156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91362c3-b88b-4543-95e9-a5bc9f53634f admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:43.313621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91362c3-b88b-4543-95e9-a5bc9f53634f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:43.314061 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91362c3-b88b-4543-95e9-a5bc9f53634f admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:43.314909 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91362c3-b88b-4543-95e9-a5bc9f53634f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:43.316897 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91362c3-b88b-4543-95e9-a5bc9f53634f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:43.324740 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:43.329436 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1379/2765] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:54:43 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:54:43.339186 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91362c3-b88b-4543-95e9-a5bc9f53634f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:43.349804 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91362c3-b88b-4543-95e9-a5bc9f53634f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:43.354380 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3e4fa5-ffc8-4fc3-9dae-a0454c8d3b47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:43.355062 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3e4fa5-ffc8-4fc3-9dae-a0454c8d3b47 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:43.355339 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3e4fa5-ffc8-4fc3-9dae-a0454c8d3b47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:43.355610 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3e4fa5-ffc8-4fc3-9dae-a0454c8d3b47 admin admin] PATH_INFO: `/v3/projects/3d6fedd1f0bc434d9aa2711c2693baf3/users/5414430ad9ff4791a4ff6b117ffd5253/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:43.372962 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3e4fa5-ffc8-4fc3-9dae-a0454c8d3b47 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d6fedd1f0bc434d9aa2711c2693baf3, user_id=5414430ad9ff4791a4ff6b117ffd5253, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:43.373300 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86112974-d713-427d-b2cf-97028ebfe8a1 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:43.375890 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3e4fa5-ffc8-4fc3-9dae-a0454c8d3b47 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:43.379834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cbc23d-a4f8-42bb-a381-6f441c292de5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '49a3011c103b4e48904f876f81d27fba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:43.380502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cbc23d-a4f8-42bb-a381-6f441c292de5 service placement] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:43.380785 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cbc23d-a4f8-42bb-a381-6f441c292de5 service placement] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:43.382027 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cbc23d-a4f8-42bb-a381-6f441c292de5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:43.382920 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b6cbc23d-a4f8-42bb-a381-6f441c292de5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:43.395165 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6cbc23d-a4f8-42bb-a381-6f441c292de5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:43.396750 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6cbc23d-a4f8-42bb-a381-6f441c292de5 service placement] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:43.407088 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1387/2766] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:54:43 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:54:43.487174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1380/2767] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:54:43 2022] PUT /identity/v3/projects/3d6fedd1f0bc434d9aa2711c2693baf3/users/5414430ad9ff4791a4ff6b117ffd5253/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:54:43.497155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee84dcff-8787-4266-809f-2287d1592bd6 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:43.497445 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee84dcff-8787-4266-809f-2287d1592bd6 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:43.497722 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee84dcff-8787-4266-809f-2287d1592bd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:43.531739 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-ee84dcff-8787-4266-809f-2287d1592bd6 None None] MFA Rules not processed for user `5414430ad9ff4791a4ff6b117ffd5253`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:54:43.538013 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ee84dcff-8787-4266-809f-2287d1592bd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:43.603269 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1388/2768] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:54:43 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:54:43.607772 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3e4fa5-ffc8-4fc3-9dae-a0454c8d3b47 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:43.670396 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3e4fa5-ffc8-4fc3-9dae-a0454c8d3b47 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:43.675082 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3050df28-7c75-4550-85af-a7f91976cf4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:43.675676 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3050df28-7c75-4550-85af-a7f91976cf4a admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:43.675951 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3050df28-7c75-4550-85af-a7f91976cf4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:43.676310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3050df28-7c75-4550-85af-a7f91976cf4a admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:43.676937 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3050df28-7c75-4550-85af-a7f91976cf4a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:43.678973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3050df28-7c75-4550-85af-a7f91976cf4a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:43.689909 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:43.755369 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1381/2769] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:54:43 2022] POST /identity/v3/projects => generated 385 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 09:54:43.759476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee84dcff-8787-4266-809f-2287d1592bd6 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:43.770875 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee84dcff-8787-4266-809f-2287d1592bd6 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:43.825278 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c80140-ebdc-44b1-8572-9849e9c30a82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:43.825880 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c80140-ebdc-44b1-8572-9849e9c30a82 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:43.826181 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c80140-ebdc-44b1-8572-9849e9c30a82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:43.826477 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c80140-ebdc-44b1-8572-9849e9c30a82 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:43.827131 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c80140-ebdc-44b1-8572-9849e9c30a82 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:43.829088 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c80140-ebdc-44b1-8572-9849e9c30a82 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:43.922262 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1389/2770] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:54:43 2022] POST /identity/v3/users => generated 299 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:54:43.923947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3050df28-7c75-4550-85af-a7f91976cf4a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:43.934114 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3050df28-7c75-4550-85af-a7f91976cf4a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:43.938692 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d81c09-fde1-402d-bd08-679bc1cdff77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:43.939279 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d81c09-fde1-402d-bd08-679bc1cdff77 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:43.939557 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d81c09-fde1-402d-bd08-679bc1cdff77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:43.939829 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d81c09-fde1-402d-bd08-679bc1cdff77 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:43.940459 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d81c09-fde1-402d-bd08-679bc1cdff77 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:43.942577 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d81c09-fde1-402d-bd08-679bc1cdff77 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:43.952366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1382/2771] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:54:43 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:54:43.956944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c80140-ebdc-44b1-8572-9849e9c30a82 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:43.966765 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c80140-ebdc-44b1-8572-9849e9c30a82 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:43.971329 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac7ef18-fc5d-4411-aec8-61c79e1e01d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:43.971956 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac7ef18-fc5d-4411-aec8-61c79e1e01d8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:43.972232 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac7ef18-fc5d-4411-aec8-61c79e1e01d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:43.972503 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac7ef18-fc5d-4411-aec8-61c79e1e01d8 admin admin] PATH_INFO: `/v3/projects/f51afa50ac314b9495ecdc8141506d89/users/9a72a78ddd0c401daae77d32aadf24b6/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:43.988331 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac7ef18-fc5d-4411-aec8-61c79e1e01d8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f51afa50ac314b9495ecdc8141506d89, user_id=9a72a78ddd0c401daae77d32aadf24b6, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:43.990998 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac7ef18-fc5d-4411-aec8-61c79e1e01d8 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:44.058247 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1390/2772] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:54:43 2022] PUT /identity/v3/projects/f51afa50ac314b9495ecdc8141506d89/users/9a72a78ddd0c401daae77d32aadf24b6/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:44.062276 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d81c09-fde1-402d-bd08-679bc1cdff77 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:44.073197 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d81c09-fde1-402d-bd08-679bc1cdff77 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:44.139791 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46aed71b-0208-4eb6-838a-e7401ef98f5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:44.140554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aed71b-0208-4eb6-838a-e7401ef98f5d admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:44.140836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aed71b-0208-4eb6-838a-e7401ef98f5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:44.141126 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aed71b-0208-4eb6-838a-e7401ef98f5d admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:44.141752 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46aed71b-0208-4eb6-838a-e7401ef98f5d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:44.143439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46aed71b-0208-4eb6-838a-e7401ef98f5d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:44.157842 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1383/2773] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:54:44 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 30 09:54:44.159412 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac7ef18-fc5d-4411-aec8-61c79e1e01d8 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:44.170795 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac7ef18-fc5d-4411-aec8-61c79e1e01d8 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:44.176626 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020c1c67-702a-4352-bc96-c57b3ef15316 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:44.177306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020c1c67-702a-4352-bc96-c57b3ef15316 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:44.177656 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020c1c67-702a-4352-bc96-c57b3ef15316 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:44.177935 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020c1c67-702a-4352-bc96-c57b3ef15316 admin admin] PATH_INFO: `/v3/projects/f51afa50ac314b9495ecdc8141506d89/users/9a72a78ddd0c401daae77d32aadf24b6/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:44.180340 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020c1c67-702a-4352-bc96-c57b3ef15316 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f51afa50ac314b9495ecdc8141506d89, user_id=9a72a78ddd0c401daae77d32aadf24b6, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:44.183044 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020c1c67-702a-4352-bc96-c57b3ef15316 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:44.186320 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:44.194891 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1391/2774] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:54:44 2022] PUT /identity/v3/projects/f51afa50ac314b9495ecdc8141506d89/users/9a72a78ddd0c401daae77d32aadf24b6/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:44.204668 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a0ff90-5dac-451b-983a-e39f0f528901 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:44.206489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a0ff90-5dac-451b-983a-e39f0f528901 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:44.206489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a0ff90-5dac-451b-983a-e39f0f528901 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:44.234544 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-01a0ff90-5dac-451b-983a-e39f0f528901 None None] MFA Rules not processed for user `9a72a78ddd0c401daae77d32aadf24b6`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:54:44.236402 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-01a0ff90-5dac-451b-983a-e39f0f528901 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:44.308127 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1384/2775] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:54:44 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:54:44.320291 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020c1c67-702a-4352-bc96-c57b3ef15316 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:44.321997 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-020c1c67-702a-4352-bc96-c57b3ef15316 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:44.378685 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020c1c67-702a-4352-bc96-c57b3ef15316 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:44.382567 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7428261b-3559-469a-9bfd-d099c04e65e4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:44.383200 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7428261b-3559-469a-9bfd-d099c04e65e4 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:44.383491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7428261b-3559-469a-9bfd-d099c04e65e4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:44.383755 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7428261b-3559-469a-9bfd-d099c04e65e4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:44.440653 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7428261b-3559-469a-9bfd-d099c04e65e4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:44.442100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7428261b-3559-469a-9bfd-d099c04e65e4 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:44.450973 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1392/2776] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:54:44 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:54:45.291208 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7472505e-ec08-444f-918d-1dd48a8cdf71 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:45.291687 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7472505e-ec08-444f-918d-1dd48a8cdf71 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:45.291791 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7472505e-ec08-444f-918d-1dd48a8cdf71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:45.310883 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7472505e-ec08-444f-918d-1dd48a8cdf71 None None] MFA Rules not processed for user `c1df4fe7054840feac0a648ce82eea1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:54:45.313416 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7472505e-ec08-444f-918d-1dd48a8cdf71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:45.366172 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1385/2777] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:54:45 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:54:46.335564 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7428261b-3559-469a-9bfd-d099c04e65e4 service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:46.345298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7428261b-3559-469a-9bfd-d099c04e65e4 service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:46.349899 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19881cd1-ac59-4a3b-a178-ebf157515724 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:46.350544 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19881cd1-ac59-4a3b-a178-ebf157515724 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:46.350814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19881cd1-ac59-4a3b-a178-ebf157515724 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:46.351077 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19881cd1-ac59-4a3b-a178-ebf157515724 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:46.412742 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19881cd1-ac59-4a3b-a178-ebf157515724 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:46.414210 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19881cd1-ac59-4a3b-a178-ebf157515724 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:46.425137 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1393/2778] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:54:46 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:54:47.949813 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7472505e-ec08-444f-918d-1dd48a8cdf71 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:47.959944 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7472505e-ec08-444f-918d-1dd48a8cdf71 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:47.963668 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982b5aeb-bec9-491d-abb1-720c57529247 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:47.964033 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982b5aeb-bec9-491d-abb1-720c57529247 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:47.964312 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982b5aeb-bec9-491d-abb1-720c57529247 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:47.964584 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982b5aeb-bec9-491d-abb1-720c57529247 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:47.970307 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:47.974524 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982b5aeb-bec9-491d-abb1-720c57529247 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:47.976020 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982b5aeb-bec9-491d-abb1-720c57529247 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:47.985476 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1386/2779] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:54:47 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:54:52.708645 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19881cd1-ac59-4a3b-a178-ebf157515724 service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:52.755749 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=100 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:52.769053 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19881cd1-ac59-4a3b-a178-ebf157515724 service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:52.774478 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9072a979-fee8-42e2-94fb-04c35432619a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:52.775144 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9072a979-fee8-42e2-94fb-04c35432619a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:52.775447 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9072a979-fee8-42e2-94fb-04c35432619a admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:52.775721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9072a979-fee8-42e2-94fb-04c35432619a admin admin] PATH_INFO: `/v3/users/35e97cc914e4413fa4af8bcf4a3d5fae` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:52.776856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9072a979-fee8-42e2-94fb-04c35432619a admin admin] RBAC: Authorizing `identity:delete_user(user_id=35e97cc914e4413fa4af8bcf4a3d5fae)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:52.778813 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9072a979-fee8-42e2-94fb-04c35432619a admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:52.940205 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-9072a979-fee8-42e2-94fb-04c35432619a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '35e97cc914e4413fa4af8bcf4a3d5fae'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:52.952456 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-9072a979-fee8-42e2-94fb-04c35432619a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '35e97cc914e4413fa4af8bcf4a3d5fae'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:52.953110 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-9072a979-fee8-42e2-94fb-04c35432619a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '35e97cc914e4413fa4af8bcf4a3d5fae'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:53.010600 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-9072a979-fee8-42e2-94fb-04c35432619a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '35e97cc914e4413fa4af8bcf4a3d5fae'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:53.017769 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-9072a979-fee8-42e2-94fb-04c35432619a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '35e97cc914e4413fa4af8bcf4a3d5fae'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:53.023809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1394/2780] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:54:52 2022] DELETE /identity/v3/users/35e97cc914e4413fa4af8bcf4a3d5fae => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:53.028280 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae2c274-f7e5-4238-97c9-e96780748d25 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:53.028746 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae2c274-f7e5-4238-97c9-e96780748d25 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:53.028933 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae2c274-f7e5-4238-97c9-e96780748d25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:53.047656 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-3ae2c274-f7e5-4238-97c9-e96780748d25 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:54:53.048862 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3ae2c274-f7e5-4238-97c9-e96780748d25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:53.105958 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1387/2781] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:54:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:54:53.116192 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9072a979-fee8-42e2-94fb-04c35432619a admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:53.117590 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-9072a979-fee8-42e2-94fb-04c35432619a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:53.175313 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9072a979-fee8-42e2-94fb-04c35432619a admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:53.179557 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf7db37-d538-40c8-91b0-dc353d68868f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:53.180164 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf7db37-d538-40c8-91b0-dc353d68868f service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:53.180449 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf7db37-d538-40c8-91b0-dc353d68868f service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:53.180725 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf7db37-d538-40c8-91b0-dc353d68868f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:53.190100 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf7db37-d538-40c8-91b0-dc353d68868f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:53.192269 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf7db37-d538-40c8-91b0-dc353d68868f service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:53.202517 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1395/2782] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:54:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:54:53.682318 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae2c274-f7e5-4238-97c9-e96780748d25 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:53.683787 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3ae2c274-f7e5-4238-97c9-e96780748d25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:53.696214 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae2c274-f7e5-4238-97c9-e96780748d25 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:53.701136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95acd98-5f96-4e51-a231-9be5c54e7f29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:53.701783 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95acd98-5f96-4e51-a231-9be5c54e7f29 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:53.702061 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95acd98-5f96-4e51-a231-9be5c54e7f29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:53.702335 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95acd98-5f96-4e51-a231-9be5c54e7f29 admin admin] PATH_INFO: `/v3/projects/9512c7b73e85419a86ba54d674de615a` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:53.703489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95acd98-5f96-4e51-a231-9be5c54e7f29 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9512c7b73e85419a86ba54d674de615a)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:53.712980 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95acd98-5f96-4e51-a231-9be5c54e7f29 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:53.813160 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b95acd98-5f96-4e51-a231-9be5c54e7f29 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9512c7b73e85419a86ba54d674de615a', 'request_id': 'req-b95acd98-5f96-4e51-a231-9be5c54e7f29'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:53.819182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b95acd98-5f96-4e51-a231-9be5c54e7f29 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9512c7b73e85419a86ba54d674de615a', 'request_id': 'req-b95acd98-5f96-4e51-a231-9be5c54e7f29'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:53.825530 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b95acd98-5f96-4e51-a231-9be5c54e7f29 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9512c7b73e85419a86ba54d674de615a', 'request_id': 'req-b95acd98-5f96-4e51-a231-9be5c54e7f29'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:53.908533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b95acd98-5f96-4e51-a231-9be5c54e7f29 admin admin] The token cache is being invalidate because project 9512c7b73e85419a86ba54d674de615a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:54:53.908824 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b95acd98-5f96-4e51-a231-9be5c54e7f29 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:53.909430 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b95acd98-5f96-4e51-a231-9be5c54e7f29 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:53.911602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1388/2783] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:54:53 2022] DELETE /identity/v3/projects/9512c7b73e85419a86ba54d674de615a => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:53.915614 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf7db37-d538-40c8-91b0-dc353d68868f service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:53.925408 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf7db37-d538-40c8-91b0-dc353d68868f service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:53.980154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ddbde7-a069-4130-8b91-7c3e5e262a90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:53.980765 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ddbde7-a069-4130-8b91-7c3e5e262a90 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:53.981040 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ddbde7-a069-4130-8b91-7c3e5e262a90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:53.981308 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ddbde7-a069-4130-8b91-7c3e5e262a90 admin admin] PATH_INFO: `/v3/users/5414430ad9ff4791a4ff6b117ffd5253` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:53.982399 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ddbde7-a069-4130-8b91-7c3e5e262a90 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5414430ad9ff4791a4ff6b117ffd5253)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:53.984365 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ddbde7-a069-4130-8b91-7c3e5e262a90 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:53.996301 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:54.196257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e9ddbde7-a069-4130-8b91-7c3e5e262a90 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5414430ad9ff4791a4ff6b117ffd5253'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:54.209590 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e9ddbde7-a069-4130-8b91-7c3e5e262a90 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5414430ad9ff4791a4ff6b117ffd5253'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:54.210268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e9ddbde7-a069-4130-8b91-7c3e5e262a90 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5414430ad9ff4791a4ff6b117ffd5253'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:54.280827 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e9ddbde7-a069-4130-8b91-7c3e5e262a90 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5414430ad9ff4791a4ff6b117ffd5253'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:54.287952 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-e9ddbde7-a069-4130-8b91-7c3e5e262a90 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5414430ad9ff4791a4ff6b117ffd5253'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:54.294003 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1396/2784] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:54:53 2022] DELETE /identity/v3/users/5414430ad9ff4791a4ff6b117ffd5253 => generated 0 bytes in 379 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:54.772633 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95acd98-5f96-4e51-a231-9be5c54e7f29 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:54.775470 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b95acd98-5f96-4e51-a231-9be5c54e7f29 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:54.788371 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95acd98-5f96-4e51-a231-9be5c54e7f29 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:54.845417 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b093d8a-6522-4872-840f-3f7ac1e3ed73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:54.846048 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b093d8a-6522-4872-840f-3f7ac1e3ed73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:54.846333 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b093d8a-6522-4872-840f-3f7ac1e3ed73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:54.846632 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b093d8a-6522-4872-840f-3f7ac1e3ed73 admin admin] PATH_INFO: `/v3/projects/3d6fedd1f0bc434d9aa2711c2693baf3` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:54.848273 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b093d8a-6522-4872-840f-3f7ac1e3ed73 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3d6fedd1f0bc434d9aa2711c2693baf3)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:54.850173 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b093d8a-6522-4872-840f-3f7ac1e3ed73 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:54.871297 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:54:54.937750 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0b093d8a-6522-4872-840f-3f7ac1e3ed73 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3d6fedd1f0bc434d9aa2711c2693baf3', 'request_id': 'req-0b093d8a-6522-4872-840f-3f7ac1e3ed73'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:54.943253 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0b093d8a-6522-4872-840f-3f7ac1e3ed73 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3d6fedd1f0bc434d9aa2711c2693baf3', 'request_id': 'req-0b093d8a-6522-4872-840f-3f7ac1e3ed73'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:54.949521 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0b093d8a-6522-4872-840f-3f7ac1e3ed73 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3d6fedd1f0bc434d9aa2711c2693baf3', 'request_id': 'req-0b093d8a-6522-4872-840f-3f7ac1e3ed73'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:55.023591 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0b093d8a-6522-4872-840f-3f7ac1e3ed73 admin admin] The token cache is being invalidate because project 3d6fedd1f0bc434d9aa2711c2693baf3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:54:55.023893 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0b093d8a-6522-4872-840f-3f7ac1e3ed73 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:55.024626 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0b093d8a-6522-4872-840f-3f7ac1e3ed73 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:55.026080 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1389/2785] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:54:54 2022] DELETE /identity/v3/projects/3d6fedd1f0bc434d9aa2711c2693baf3 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:55.030019 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ddbde7-a069-4130-8b91-7c3e5e262a90 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:55.031578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-e9ddbde7-a069-4130-8b91-7c3e5e262a90 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:55.048787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ddbde7-a069-4130-8b91-7c3e5e262a90 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:55.100847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd491ab-93e5-4b59-81d5-5f8f54056242 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:55.101469 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd491ab-93e5-4b59-81d5-5f8f54056242 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:55.101750 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd491ab-93e5-4b59-81d5-5f8f54056242 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:55.102028 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd491ab-93e5-4b59-81d5-5f8f54056242 admin admin] PATH_INFO: `/v3/users/9a72a78ddd0c401daae77d32aadf24b6` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:55.103157 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd491ab-93e5-4b59-81d5-5f8f54056242 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9a72a78ddd0c401daae77d32aadf24b6)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:55.105277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd491ab-93e5-4b59-81d5-5f8f54056242 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:55.312476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8cd491ab-93e5-4b59-81d5-5f8f54056242 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9a72a78ddd0c401daae77d32aadf24b6'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:55.324308 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8cd491ab-93e5-4b59-81d5-5f8f54056242 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9a72a78ddd0c401daae77d32aadf24b6'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:55.324966 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8cd491ab-93e5-4b59-81d5-5f8f54056242 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9a72a78ddd0c401daae77d32aadf24b6'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:55.372427 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8cd491ab-93e5-4b59-81d5-5f8f54056242 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9a72a78ddd0c401daae77d32aadf24b6'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:55.380241 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8cd491ab-93e5-4b59-81d5-5f8f54056242 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9a72a78ddd0c401daae77d32aadf24b6'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:55.386083 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1397/2786] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:54:55 2022] DELETE /identity/v3/users/9a72a78ddd0c401daae77d32aadf24b6 => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:54:55.841370 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b093d8a-6522-4872-840f-3f7ac1e3ed73 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:54:55.844703 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0b093d8a-6522-4872-840f-3f7ac1e3ed73 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:54:55.856359 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b093d8a-6522-4872-840f-3f7ac1e3ed73 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:54:55.915045 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc3e102-3ee4-4d02-9731-4fffe777f725 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:54:55.915686 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc3e102-3ee4-4d02-9731-4fffe777f725 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:54:55.915965 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc3e102-3ee4-4d02-9731-4fffe777f725 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:54:55.916236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc3e102-3ee4-4d02-9731-4fffe777f725 admin admin] PATH_INFO: `/v3/projects/f51afa50ac314b9495ecdc8141506d89` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:54:55.917330 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc3e102-3ee4-4d02-9731-4fffe777f725 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f51afa50ac314b9495ecdc8141506d89)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:54:55.919308 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc3e102-3ee4-4d02-9731-4fffe777f725 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:54:56.012492 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7fc3e102-3ee4-4d02-9731-4fffe777f725 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f51afa50ac314b9495ecdc8141506d89', 'request_id': 'req-7fc3e102-3ee4-4d02-9731-4fffe777f725'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:56.018137 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7fc3e102-3ee4-4d02-9731-4fffe777f725 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f51afa50ac314b9495ecdc8141506d89', 'request_id': 'req-7fc3e102-3ee4-4d02-9731-4fffe777f725'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:56.024292 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7fc3e102-3ee4-4d02-9731-4fffe777f725 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f51afa50ac314b9495ecdc8141506d89', 'request_id': 'req-7fc3e102-3ee4-4d02-9731-4fffe777f725'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:56.080504 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7fc3e102-3ee4-4d02-9731-4fffe777f725 admin admin] The token cache is being invalidate because project f51afa50ac314b9495ecdc8141506d89 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:54:56.080790 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7fc3e102-3ee4-4d02-9731-4fffe777f725 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:56.081399 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7fc3e102-3ee4-4d02-9731-4fffe777f725 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:54:56.082933 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1390/2787] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:54:55 2022] DELETE /identity/v3/projects/f51afa50ac314b9495ecdc8141506d89 => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:55:05.382751 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=75,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:55:06.077135 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=67,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:55:08.601999 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54a04d5-201d-4684-8748-c23553df00cd None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:55:08.602483 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54a04d5-201d-4684-8748-c23553df00cd None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:55:08.602615 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54a04d5-201d-4684-8748-c23553df00cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:55:08.623873 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-a54a04d5-201d-4684-8748-c23553df00cd None None] MFA Rules not processed for user `21d5e60badb846fb840a65f443728a9b`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:55:08.627037 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a54a04d5-201d-4684-8748-c23553df00cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:55:08.679271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1398/2788] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:55:08 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:55:18.671881 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=25 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:56:01.699688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc3e102-3ee4-4d02-9731-4fffe777f725 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:56:01.701135 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7fc3e102-3ee4-4d02-9731-4fffe777f725 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:56:01.758753 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc3e102-3ee4-4d02-9731-4fffe777f725 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:56:01.762338 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6edf19-a014-45b3-8f67-18118a9b745b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf907d4033064fb8a2707b1bbb0a180b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:56:01.762979 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6edf19-a014-45b3-8f67-18118a9b745b service glance] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:56:01.763265 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6edf19-a014-45b3-8f67-18118a9b745b service glance] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:56:01.763538 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6edf19-a014-45b3-8f67-18118a9b745b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:56:01.765254 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ed6edf19-a014-45b3-8f67-18118a9b745b service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:56:01.846431 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6edf19-a014-45b3-8f67-18118a9b745b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:56:01.847885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6edf19-a014-45b3-8f67-18118a9b745b service glance] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:56:01.867620 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1391/2789] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:56:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:56:02.095243 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54a04d5-201d-4684-8748-c23553df00cd None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:56:02.096738 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-a54a04d5-201d-4684-8748-c23553df00cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:56:02.154172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54a04d5-201d-4684-8748-c23553df00cd None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:56:02.157867 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d920fb1b-d246-4b6b-9c14-03e9553e5aa8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:56:02.158466 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d920fb1b-d246-4b6b-9c14-03e9553e5aa8 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:56:02.158779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d920fb1b-d246-4b6b-9c14-03e9553e5aa8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:56:02.159051 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d920fb1b-d246-4b6b-9c14-03e9553e5aa8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:56:02.160762 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d920fb1b-d246-4b6b-9c14-03e9553e5aa8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:56:02.174024 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d920fb1b-d246-4b6b-9c14-03e9553e5aa8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:56:02.175502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d920fb1b-d246-4b6b-9c14-03e9553e5aa8 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:56:02.187128 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1399/2790] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:56:02 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:56:11.865105 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=63 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:56:12.182079 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=26 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:56:20.859535 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9453d031-04d5-4c26-b2fb-2605f9b0f260 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:56:20.860012 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9453d031-04d5-4c26-b2fb-2605f9b0f260 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:56:20.860151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9453d031-04d5-4c26-b2fb-2605f9b0f260 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:56:20.910090 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9453d031-04d5-4c26-b2fb-2605f9b0f260 None None] MFA Rules not processed for user `7d6e0545a40a436d8c63820b795551a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:56:20.911688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9453d031-04d5-4c26-b2fb-2605f9b0f260 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:56:20.979191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1392/2791] 10.222.0.39 () {58 vars in 1009 bytes} [Fri Sep 30 09:56:20 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:56:21.107366 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d920fb1b-d246-4b6b-9c14-03e9553e5aa8 service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:56:21.108824 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d920fb1b-d246-4b6b-9c14-03e9553e5aa8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:56:21.198280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d920fb1b-d246-4b6b-9c14-03e9553e5aa8 service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:56:21.203550 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdda174-fb3f-4d91-ad44-2c4534b17dba service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e5ad921e6fa41c297696ec7279f0fc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:56:21.204152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdda174-fb3f-4d91-ad44-2c4534b17dba service swift] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:56:21.204434 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdda174-fb3f-4d91-ad44-2c4534b17dba service swift] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:56:21.204708 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdda174-fb3f-4d91-ad44-2c4534b17dba service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:56:21.212895 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdda174-fb3f-4d91-ad44-2c4534b17dba service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:56:21.214482 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdda174-fb3f-4d91-ad44-2c4534b17dba service swift] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:56:21.222369 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1400/2792] 10.222.0.39 () {60 vars in 1384 bytes} [Fri Sep 30 09:56:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 30 09:56:30.975933 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=40 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:56:31.219203 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=41 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:56:37.748703 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310af8dd-bfa2-4640-8b84-700d4c4ab378 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:56:37.749309 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310af8dd-bfa2-4640-8b84-700d4c4ab378 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:56:37.749309 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310af8dd-bfa2-4640-8b84-700d4c4ab378 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:56:37.751177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1393/2793] 10.222.0.39 () {54 vars in 921 bytes} [Fri Sep 30 09:56:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 30 09:56:37.755535 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b066a605-9440-4290-8be9-3ece772d27be None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:56:37.756235 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b066a605-9440-4290-8be9-3ece772d27be None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:56:37.756463 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b066a605-9440-4290-8be9-3ece772d27be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:56:37.797296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-b066a605-9440-4290-8be9-3ece772d27be None None] MFA Rules not processed for user `7ff68fb87700423ab53c90520a6bd3b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:56:37.798532 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b066a605-9440-4290-8be9-3ece772d27be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:56:37.807849 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1401/2794] 10.222.0.39 () {58 vars in 1018 bytes} [Fri Sep 30 09:56:37 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 09:56:37.819221 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310af8dd-bfa2-4640-8b84-700d4c4ab378 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:56:37.830559 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310af8dd-bfa2-4640-8b84-700d4c4ab378 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:56:37.834674 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bd781f-7d3b-4c42-a984-f22d4533b7bb service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf907d4033064fb8a2707b1bbb0a180b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:56:37.835314 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bd781f-7d3b-4c42-a984-f22d4533b7bb service glance] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:56:37.835596 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bd781f-7d3b-4c42-a984-f22d4533b7bb service glance] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:56:37.835871 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bd781f-7d3b-4c42-a984-f22d4533b7bb service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:56:37.846905 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bd781f-7d3b-4c42-a984-f22d4533b7bb service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:56:37.848326 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bd781f-7d3b-4c42-a984-f22d4533b7bb service glance] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:56:37.858935 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1394/2795] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:56:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:56:40.443204 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5a4b8f-b1fd-407e-a932-c5db2445db7a None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:56:40.443683 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5a4b8f-b1fd-407e-a932-c5db2445db7a None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:56:40.443888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5a4b8f-b1fd-407e-a932-c5db2445db7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:56:40.463049 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-1c5a4b8f-b1fd-407e-a932-c5db2445db7a None None] MFA Rules not processed for user `7d6e0545a40a436d8c63820b795551a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:56:40.464261 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-1c5a4b8f-b1fd-407e-a932-c5db2445db7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:56:40.471290 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1402/2796] 10.222.0.39 () {58 vars in 1009 bytes} [Fri Sep 30 09:56:40 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:56:40.483608 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bd781f-7d3b-4c42-a984-f22d4533b7bb service glance] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:56:40.494784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bd781f-7d3b-4c42-a984-f22d4533b7bb service glance] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:56:40.499754 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1ca012-a10c-4892-ac27-13c667501483 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e5ad921e6fa41c297696ec7279f0fc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:56:40.500344 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1ca012-a10c-4892-ac27-13c667501483 service swift] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:56:40.500652 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1ca012-a10c-4892-ac27-13c667501483 service swift] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:56:40.500933 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1ca012-a10c-4892-ac27-13c667501483 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:56:40.509030 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1ca012-a10c-4892-ac27-13c667501483 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:56:40.510693 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1ca012-a10c-4892-ac27-13c667501483 service swift] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:56:40.518551 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1395/2797] 10.222.0.39 () {60 vars in 1384 bytes} [Fri Sep 30 09:56:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 30 09:56:45.181534 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5a4b8f-b1fd-407e-a932-c5db2445db7a None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:56:45.183104 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-1c5a4b8f-b1fd-407e-a932-c5db2445db7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:56:45.243295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5a4b8f-b1fd-407e-a932-c5db2445db7a None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:56:45.246995 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeca5373-be57-475c-870d-a047970ae190 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:56:45.247611 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeca5373-be57-475c-870d-a047970ae190 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:56:45.247894 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeca5373-be57-475c-870d-a047970ae190 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:56:45.248194 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeca5373-be57-475c-870d-a047970ae190 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:56:45.250092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-aeca5373-be57-475c-870d-a047970ae190 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:56:45.315597 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeca5373-be57-475c-870d-a047970ae190 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:56:45.317066 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeca5373-be57-475c-870d-a047970ae190 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:56:45.328681 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1403/2798] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:56:45 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:56:50.515393 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=12 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:56:54.703004 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adde056a-de96-4ea8-a3fd-8021c6b1642c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:56:54.703621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adde056a-de96-4ea8-a3fd-8021c6b1642c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:56:54.703621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adde056a-de96-4ea8-a3fd-8021c6b1642c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:56:54.722399 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-adde056a-de96-4ea8-a3fd-8021c6b1642c None None] MFA Rules not processed for user `c1df4fe7054840feac0a648ce82eea1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:56:54.723656 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-adde056a-de96-4ea8-a3fd-8021c6b1642c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:56:54.730448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1396/2799] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:56:54 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:56:55.328733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=60 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:56:55.714295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeca5373-be57-475c-870d-a047970ae190 service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:56:55.717013 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-aeca5373-be57-475c-870d-a047970ae190 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:56:55.802892 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeca5373-be57-475c-870d-a047970ae190 service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:56:55.807904 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4e8906-00b7-447c-94e9-b3086afae359 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '49a3011c103b4e48904f876f81d27fba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:56:55.808460 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4e8906-00b7-447c-94e9-b3086afae359 service placement] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:56:55.808741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4e8906-00b7-447c-94e9-b3086afae359 service placement] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:56:55.809015 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4e8906-00b7-447c-94e9-b3086afae359 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:56:55.810749 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-8c4e8906-00b7-447c-94e9-b3086afae359 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:56:55.822831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4e8906-00b7-447c-94e9-b3086afae359 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:56:55.824415 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4e8906-00b7-447c-94e9-b3086afae359 service placement] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:56:55.834420 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1404/2800] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:56:55 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:56:56.165285 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068b7454-2163-4371-9e36-c794525b6295 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:56:56.165816 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068b7454-2163-4371-9e36-c794525b6295 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:56:56.166005 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068b7454-2163-4371-9e36-c794525b6295 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:56:56.187329 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-068b7454-2163-4371-9e36-c794525b6295 None None] MFA Rules not processed for user `7d6e0545a40a436d8c63820b795551a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:56:56.188531 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-068b7454-2163-4371-9e36-c794525b6295 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:56:56.196120 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1397/2801] 10.222.0.39 () {58 vars in 967 bytes} [Fri Sep 30 09:56:56 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:56:56.209612 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4e8906-00b7-447c-94e9-b3086afae359 service placement] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:56:56.221049 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4e8906-00b7-447c-94e9-b3086afae359 service placement] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:56:56.226166 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a99203f-26cd-4a53-94bf-fb6455cf5b14 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0e5ad921e6fa41c297696ec7279f0fc4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:56:56.226802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a99203f-26cd-4a53-94bf-fb6455cf5b14 service swift] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:56:56.227083 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a99203f-26cd-4a53-94bf-fb6455cf5b14 service swift] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:56:56.227373 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a99203f-26cd-4a53-94bf-fb6455cf5b14 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:56:56.235395 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a99203f-26cd-4a53-94bf-fb6455cf5b14 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:56:56.237086 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a99203f-26cd-4a53-94bf-fb6455cf5b14 service swift] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:56:56.245987 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1405/2802] 10.222.0.39 () {60 vars in 1384 bytes} [Fri Sep 30 09:56:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 30 09:56:59.745352 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068b7454-2163-4371-9e36-c794525b6295 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:56:59.747422 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-068b7454-2163-4371-9e36-c794525b6295 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:56:59.764406 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068b7454-2163-4371-9e36-c794525b6295 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:56:59.770016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7e69b7-cf4b-4fcf-962f-ee2c18e81276 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:56:59.770830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7e69b7-cf4b-4fcf-962f-ee2c18e81276 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:56:59.771113 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7e69b7-cf4b-4fcf-962f-ee2c18e81276 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:56:59.771391 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7e69b7-cf4b-4fcf-962f-ee2c18e81276 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:56:59.773115 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4b7e69b7-cf4b-4fcf-962f-ee2c18e81276 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:56:59.786404 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7e69b7-cf4b-4fcf-962f-ee2c18e81276 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:56:59.788123 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7e69b7-cf4b-4fcf-962f-ee2c18e81276 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:56:59.799129 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1398/2803] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:56:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:57:06.241980 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=38 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:57:09.114046 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a99203f-26cd-4a53-94bf-fb6455cf5b14 service swift] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:57:09.115583 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0a99203f-26cd-4a53-94bf-fb6455cf5b14 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:57:09.127524 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a99203f-26cd-4a53-94bf-fb6455cf5b14 service swift] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:57:09.131336 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc684212-af90-47a4-af07-2cd2da1fedef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:57:09.131919 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc684212-af90-47a4-af07-2cd2da1fedef service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:09.132209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc684212-af90-47a4-af07-2cd2da1fedef service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:09.132464 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc684212-af90-47a4-af07-2cd2da1fedef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:09.134092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-bc684212-af90-47a4-af07-2cd2da1fedef service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:57:09.143383 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc684212-af90-47a4-af07-2cd2da1fedef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:57:09.144886 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc684212-af90-47a4-af07-2cd2da1fedef service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:57:09.153286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1406/2804] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:57:09 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:57:09.794631 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=12 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:57:12.411558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7e69b7-cf4b-4fcf-962f-ee2c18e81276 service nova] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:57:12.422748 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7e69b7-cf4b-4fcf-962f-ee2c18e81276 service nova] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:57:12.428218 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3cde1c-c279-4027-bc7f-82ac0f20cc7e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:57:12.428807 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3cde1c-c279-4027-bc7f-82ac0f20cc7e service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:12.429083 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3cde1c-c279-4027-bc7f-82ac0f20cc7e service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:12.429351 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3cde1c-c279-4027-bc7f-82ac0f20cc7e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:12.431065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9f3cde1c-c279-4027-bc7f-82ac0f20cc7e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:57:12.442830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3cde1c-c279-4027-bc7f-82ac0f20cc7e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:57:12.444385 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3cde1c-c279-4027-bc7f-82ac0f20cc7e service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:57:12.457183 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1399/2805] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:57:12 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:57:16.384415 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc684212-af90-47a4-af07-2cd2da1fedef service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:57:16.397543 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc684212-af90-47a4-af07-2cd2da1fedef service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:57:16.403717 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f66bda4-5051-47bf-beab-e09c697c4772 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:57:16.404325 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f66bda4-5051-47bf-beab-e09c697c4772 service nova] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:16.404603 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f66bda4-5051-47bf-beab-e09c697c4772 service nova] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:16.404877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f66bda4-5051-47bf-beab-e09c697c4772 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:16.406634 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-3f66bda4-5051-47bf-beab-e09c697c4772 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:57:16.419947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f66bda4-5051-47bf-beab-e09c697c4772 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:57:16.421436 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f66bda4-5051-47bf-beab-e09c697c4772 service nova] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:57:16.432322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1407/2806] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:57:16 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:57:22.450893 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=6 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:57:26.427636 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=12 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:57:30.421527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3cde1c-c279-4027-bc7f-82ac0f20cc7e service nova] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:57:30.430791 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3cde1c-c279-4027-bc7f-82ac0f20cc7e service nova] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:57:30.434286 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12c84ea-c282-4fd8-9411-64f2bd7cb890 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:57:30.434898 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12c84ea-c282-4fd8-9411-64f2bd7cb890 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:30.435177 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12c84ea-c282-4fd8-9411-64f2bd7cb890 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:30.435448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12c84ea-c282-4fd8-9411-64f2bd7cb890 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:30.437143 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f12c84ea-c282-4fd8-9411-64f2bd7cb890 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:57:30.447468 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12c84ea-c282-4fd8-9411-64f2bd7cb890 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:57:30.449417 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12c84ea-c282-4fd8-9411-64f2bd7cb890 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:57:30.458057 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1400/2807] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:57:30 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:57:39.998825 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f66bda4-5051-47bf-beab-e09c697c4772 service nova] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:57:40.008451 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f66bda4-5051-47bf-beab-e09c697c4772 service nova] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:57:40.012091 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c55376d-d70d-4f97-bd4a-641bc82da5aa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:57:40.012678 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c55376d-d70d-4f97-bd4a-641bc82da5aa service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:40.012959 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c55376d-d70d-4f97-bd4a-641bc82da5aa service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:40.013233 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c55376d-d70d-4f97-bd4a-641bc82da5aa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:40.015006 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-8c55376d-d70d-4f97-bd4a-641bc82da5aa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:57:40.076972 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c55376d-d70d-4f97-bd4a-641bc82da5aa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:57:40.078418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c55376d-d70d-4f97-bd4a-641bc82da5aa service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:57:40.087857 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1408/2808] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:57:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:57:40.454044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=6 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:57:43.527507 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12c84ea-c282-4fd8-9411-64f2bd7cb890 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:57:43.539547 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12c84ea-c282-4fd8-9411-64f2bd7cb890 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:57:43.544901 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7cbc98-7df1-4702-9c01-2fcbf2399592 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '49a3011c103b4e48904f876f81d27fba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:57:43.545508 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7cbc98-7df1-4702-9c01-2fcbf2399592 service placement] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:43.545790 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7cbc98-7df1-4702-9c01-2fcbf2399592 service placement] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:43.546089 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7cbc98-7df1-4702-9c01-2fcbf2399592 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:43.547832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-bf7cbc98-7df1-4702-9c01-2fcbf2399592 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:57:43.560052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf7cbc98-7df1-4702-9c01-2fcbf2399592 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:57:43.561616 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf7cbc98-7df1-4702-9c01-2fcbf2399592 service placement] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:57:43.571526 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1401/2809] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:57:43 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:57:43.648244 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c55376d-d70d-4f97-bd4a-641bc82da5aa service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:57:43.648244 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-8c55376d-d70d-4f97-bd4a-641bc82da5aa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:57:43.660310 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c55376d-d70d-4f97-bd4a-641bc82da5aa service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:57:43.664959 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990583d5-ef90-4eeb-ae9c-35a6bf5be3c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:57:43.665536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990583d5-ef90-4eeb-ae9c-35a6bf5be3c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:43.665818 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990583d5-ef90-4eeb-ae9c-35a6bf5be3c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:43.666172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990583d5-ef90-4eeb-ae9c-35a6bf5be3c9 admin admin] PATH_INFO: `/v3/users/21d5e60badb846fb840a65f443728a9b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:43.667348 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990583d5-ef90-4eeb-ae9c-35a6bf5be3c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=21d5e60badb846fb840a65f443728a9b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:57:43.669935 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990583d5-ef90-4eeb-ae9c-35a6bf5be3c9 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:57:43.869980 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-990583d5-ef90-4eeb-ae9c-35a6bf5be3c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21d5e60badb846fb840a65f443728a9b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:57:43.882930 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-990583d5-ef90-4eeb-ae9c-35a6bf5be3c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21d5e60badb846fb840a65f443728a9b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:57:43.883594 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-990583d5-ef90-4eeb-ae9c-35a6bf5be3c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21d5e60badb846fb840a65f443728a9b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:57:43.964175 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-990583d5-ef90-4eeb-ae9c-35a6bf5be3c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21d5e60badb846fb840a65f443728a9b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:57:43.971538 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-990583d5-ef90-4eeb-ae9c-35a6bf5be3c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21d5e60badb846fb840a65f443728a9b'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:57:43.977529 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1409/2810] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:57:43 2022] DELETE /identity/v3/users/21d5e60badb846fb840a65f443728a9b => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 09:57:44.229039 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7cbc98-7df1-4702-9c01-2fcbf2399592 service placement] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:57:44.240440 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7cbc98-7df1-4702-9c01-2fcbf2399592 service placement] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:57:44.296934 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a960e718-aceb-4562-9202-18e2693db664 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:57:44.297558 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a960e718-aceb-4562-9202-18e2693db664 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:44.297836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a960e718-aceb-4562-9202-18e2693db664 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:44.298106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a960e718-aceb-4562-9202-18e2693db664 admin admin] PATH_INFO: `/v3/projects/1142014cc9284cf1833e5bfbcdb41599` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:44.299242 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a960e718-aceb-4562-9202-18e2693db664 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1142014cc9284cf1833e5bfbcdb41599)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:57:44.301151 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a960e718-aceb-4562-9202-18e2693db664 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:57:44.398969 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a960e718-aceb-4562-9202-18e2693db664 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1142014cc9284cf1833e5bfbcdb41599', 'request_id': 'req-a960e718-aceb-4562-9202-18e2693db664'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:57:44.404953 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a960e718-aceb-4562-9202-18e2693db664 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1142014cc9284cf1833e5bfbcdb41599', 'request_id': 'req-a960e718-aceb-4562-9202-18e2693db664'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:57:44.411011 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a960e718-aceb-4562-9202-18e2693db664 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1142014cc9284cf1833e5bfbcdb41599', 'request_id': 'req-a960e718-aceb-4562-9202-18e2693db664'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:57:44.489392 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a960e718-aceb-4562-9202-18e2693db664 admin admin] The token cache is being invalidate because project 1142014cc9284cf1833e5bfbcdb41599 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:57:44.489615 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a960e718-aceb-4562-9202-18e2693db664 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:57:44.490229 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a960e718-aceb-4562-9202-18e2693db664 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:57:44.491713 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1402/2811] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:57:44 2022] DELETE /identity/v3/projects/1142014cc9284cf1833e5bfbcdb41599 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:57:44.496584 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ba4ddb-590c-4053-89dc-736ba430ef36 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:44.496874 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ba4ddb-590c-4053-89dc-736ba430ef36 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:44.497147 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ba4ddb-590c-4053-89dc-736ba430ef36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:44.517543 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-69ba4ddb-590c-4053-89dc-736ba430ef36 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:57:44.518726 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-69ba4ddb-590c-4053-89dc-736ba430ef36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:57:44.577390 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1410/2812] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:57:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:57:44.585149 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfa4437-49d9-4275-822d-009be00df4a9 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:44.585642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfa4437-49d9-4275-822d-009be00df4a9 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:44.585771 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfa4437-49d9-4275-822d-009be00df4a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:44.606425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9dfa4437-49d9-4275-822d-009be00df4a9 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:57:44.607736 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9dfa4437-49d9-4275-822d-009be00df4a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:57:44.615731 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1403/2813] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:57:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:57:44.619914 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ba4ddb-590c-4053-89dc-736ba430ef36 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:57:44.630748 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ba4ddb-590c-4053-89dc-736ba430ef36 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:57:44.635711 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3a0787-8c1e-4e31-9186-21133e38f221 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:57:44.636292 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3a0787-8c1e-4e31-9186-21133e38f221 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:44.636566 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3a0787-8c1e-4e31-9186-21133e38f221 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:44.637360 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3a0787-8c1e-4e31-9186-21133e38f221 admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:44.637977 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3a0787-8c1e-4e31-9186-21133e38f221 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 09:57:44.638281 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3a0787-8c1e-4e31-9186-21133e38f221 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:57:44.639945 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3a0787-8c1e-4e31-9186-21133e38f221 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:57:44.641402 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:57:44.653792 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1411/2814] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 09:57:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 09:57:44.657263 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfa4437-49d9-4275-822d-009be00df4a9 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:57:44.666973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfa4437-49d9-4275-822d-009be00df4a9 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:57:44.671625 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713e329d-88b6-4fa8-a9fb-f8242dd72520 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:57:44.672318 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713e329d-88b6-4fa8-a9fb-f8242dd72520 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:44.672593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713e329d-88b6-4fa8-a9fb-f8242dd72520 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:44.673047 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713e329d-88b6-4fa8-a9fb-f8242dd72520 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:44.673676 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713e329d-88b6-4fa8-a9fb-f8242dd72520 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:57:44.675707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713e329d-88b6-4fa8-a9fb-f8242dd72520 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:57:44.761396 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1404/2815] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 09:57:44 2022] POST /identity/v3/projects => generated 377 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:57:44.765837 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3a0787-8c1e-4e31-9186-21133e38f221 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:57:44.776022 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3a0787-8c1e-4e31-9186-21133e38f221 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:57:44.832195 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0204e6e5-ef6b-48cc-957f-78f1b74490fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:57:44.832791 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0204e6e5-ef6b-48cc-957f-78f1b74490fc admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:44.833073 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0204e6e5-ef6b-48cc-957f-78f1b74490fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:44.833353 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0204e6e5-ef6b-48cc-957f-78f1b74490fc admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:44.833971 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0204e6e5-ef6b-48cc-957f-78f1b74490fc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:57:44.835922 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0204e6e5-ef6b-48cc-957f-78f1b74490fc admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:57:44.924183 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1412/2816] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 09:57:44 2022] POST /identity/v3/users => generated 289 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 09:57:44.928250 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713e329d-88b6-4fa8-a9fb-f8242dd72520 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:57:44.939071 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713e329d-88b6-4fa8-a9fb-f8242dd72520 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:57:44.943712 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1ffdc1-8159-42e5-aef5-948b2ec6805c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:57:44.944321 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1ffdc1-8159-42e5-aef5-948b2ec6805c admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:44.944603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1ffdc1-8159-42e5-aef5-948b2ec6805c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:44.944879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1ffdc1-8159-42e5-aef5-948b2ec6805c admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:44.945513 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1ffdc1-8159-42e5-aef5-948b2ec6805c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:57:44.947182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1ffdc1-8159-42e5-aef5-948b2ec6805c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:57:44.957867 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1405/2817] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:57:44 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:57:44.961821 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0204e6e5-ef6b-48cc-957f-78f1b74490fc admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:57:44.971608 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0204e6e5-ef6b-48cc-957f-78f1b74490fc admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:57:44.976267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8887ca2-1b94-4ae7-a708-f97d98209244 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:57:44.976845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8887ca2-1b94-4ae7-a708-f97d98209244 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:44.977121 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8887ca2-1b94-4ae7-a708-f97d98209244 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:44.977392 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8887ca2-1b94-4ae7-a708-f97d98209244 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:44.978022 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8887ca2-1b94-4ae7-a708-f97d98209244 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:57:44.979640 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8887ca2-1b94-4ae7-a708-f97d98209244 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:57:44.989962 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1413/2818] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 09:57:44 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 09:57:44.993933 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1ffdc1-8159-42e5-aef5-948b2ec6805c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:57:45.004277 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1ffdc1-8159-42e5-aef5-948b2ec6805c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:57:45.009098 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8695e93-1e94-4b3a-b01e-0b300cc7d62c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:57:45.009727 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8695e93-1e94-4b3a-b01e-0b300cc7d62c admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:45.010005 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8695e93-1e94-4b3a-b01e-0b300cc7d62c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:45.010273 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8695e93-1e94-4b3a-b01e-0b300cc7d62c admin admin] PATH_INFO: `/v3/projects/85eef8fd790146cfa7da6214c13ca262/users/ee46fc4dd13143bfb409fb5033d9a4d2/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:45.025263 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8695e93-1e94-4b3a-b01e-0b300cc7d62c admin admin] RBAC: Authorizing `identity:create_grant(project_id=85eef8fd790146cfa7da6214c13ca262, user_id=ee46fc4dd13143bfb409fb5033d9a4d2, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:57:45.027899 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8695e93-1e94-4b3a-b01e-0b300cc7d62c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:57:45.089803 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1406/2819] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 09:57:44 2022] PUT /identity/v3/projects/85eef8fd790146cfa7da6214c13ca262/users/ee46fc4dd13143bfb409fb5033d9a4d2/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:57:45.095146 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e41164d-4974-4b0f-8952-5e56ef4194a1 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:45.095431 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e41164d-4974-4b0f-8952-5e56ef4194a1 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:45.095704 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e41164d-4974-4b0f-8952-5e56ef4194a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:45.114541 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-4e41164d-4974-4b0f-8952-5e56ef4194a1 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:57:45.115759 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-4e41164d-4974-4b0f-8952-5e56ef4194a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:57:45.173931 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1414/2820] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:57:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:57:45.184488 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8695e93-1e94-4b3a-b01e-0b300cc7d62c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:57:45.186199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a8695e93-1e94-4b3a-b01e-0b300cc7d62c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:57:45.193495 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:57:45.242248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8695e93-1e94-4b3a-b01e-0b300cc7d62c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:57:45.247065 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef9a3b4-829a-4845-a39a-445e7e1f9b7b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:57:45.247676 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef9a3b4-829a-4845-a39a-445e7e1f9b7b service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:45.247955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef9a3b4-829a-4845-a39a-445e7e1f9b7b service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:45.248229 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef9a3b4-829a-4845-a39a-445e7e1f9b7b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:45.257465 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef9a3b4-829a-4845-a39a-445e7e1f9b7b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:57:45.258940 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef9a3b4-829a-4845-a39a-445e7e1f9b7b service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:57:45.268486 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1407/2821] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:57:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:57:50.389219 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370229b0-c1f9-49c9-9ab3-f17addfa3b8f None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:50.389721 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370229b0-c1f9-49c9-9ab3-f17addfa3b8f None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:50.389831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370229b0-c1f9-49c9-9ab3-f17addfa3b8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:50.424122 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-370229b0-c1f9-49c9-9ab3-f17addfa3b8f None None] MFA Rules not processed for user `ee46fc4dd13143bfb409fb5033d9a4d2`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:57:50.427372 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-370229b0-c1f9-49c9-9ab3-f17addfa3b8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:57:50.502142 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1415/2822] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:57:50 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:57:50.516067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef9a3b4-829a-4845-a39a-445e7e1f9b7b service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:57:50.516067 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-eef9a3b4-829a-4845-a39a-445e7e1f9b7b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:57:50.595206 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef9a3b4-829a-4845-a39a-445e7e1f9b7b service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:57:50.600744 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a232c74-f841-4a5e-bf76-882f0e001a06 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:57:50.601366 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a232c74-f841-4a5e-bf76-882f0e001a06 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:50.601648 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a232c74-f841-4a5e-bf76-882f0e001a06 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:50.601916 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a232c74-f841-4a5e-bf76-882f0e001a06 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:50.611485 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a232c74-f841-4a5e-bf76-882f0e001a06 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:57:50.618825 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a232c74-f841-4a5e-bf76-882f0e001a06 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:57:50.623345 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1408/2823] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:57:50 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 09:57:53.006266 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46544ca-3648-4698-8e0d-023d35d3fc0a None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:53.007095 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46544ca-3648-4698-8e0d-023d35d3fc0a None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:53.007254 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46544ca-3648-4698-8e0d-023d35d3fc0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:53.015345 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:57:53.030324 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-c46544ca-3648-4698-8e0d-023d35d3fc0a None None] MFA Rules not processed for user `ee46fc4dd13143bfb409fb5033d9a4d2`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 09:57:53.031595 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-c46544ca-3648-4698-8e0d-023d35d3fc0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:57:53.038422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1416/2824] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 09:57:53 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 09:57:53.375062 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a232c74-f841-4a5e-bf76-882f0e001a06 service nova] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:57:53.385134 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a232c74-f841-4a5e-bf76-882f0e001a06 service nova] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:57:53.389311 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97708b3d-0b4f-4310-9fec-69b921ec3786 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:57:53.389908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97708b3d-0b4f-4310-9fec-69b921ec3786 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:57:53.390186 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97708b3d-0b4f-4310-9fec-69b921ec3786 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:57:53.390458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97708b3d-0b4f-4310-9fec-69b921ec3786 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:57:53.392239 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-97708b3d-0b4f-4310-9fec-69b921ec3786 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:57:53.401992 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97708b3d-0b4f-4310-9fec-69b921ec3786 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:57:53.407474 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97708b3d-0b4f-4310-9fec-69b921ec3786 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:57:53.411621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1409/2825] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:57:53 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:58:03.407871 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=64 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:58:03.427542 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:59:04.298953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46544ca-3648-4698-8e0d-023d35d3fc0a None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:59:04.309322 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46544ca-3648-4698-8e0d-023d35d3fc0a None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:59:04.313496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c48f1c1-a2bb-401c-b7e7-bb1d34a81361 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:59:04.314095 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c48f1c1-a2bb-401c-b7e7-bb1d34a81361 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:59:04.314377 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c48f1c1-a2bb-401c-b7e7-bb1d34a81361 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:59:04.314697 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c48f1c1-a2bb-401c-b7e7-bb1d34a81361 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:59:04.316422 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-5c48f1c1-a2bb-401c-b7e7-bb1d34a81361 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:59:04.328783 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c48f1c1-a2bb-401c-b7e7-bb1d34a81361 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:59:04.331900 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c48f1c1-a2bb-401c-b7e7-bb1d34a81361 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:59:04.339944 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1417/2826] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:59:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:59:11.245733 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97708b3d-0b4f-4310-9fec-69b921ec3786 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:59:11.248453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-97708b3d-0b4f-4310-9fec-69b921ec3786 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:59:11.316949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97708b3d-0b4f-4310-9fec-69b921ec3786 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:59:11.321922 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8eeb43-35ce-4f11-8bdb-c4be1f4bcc72 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '49a3011c103b4e48904f876f81d27fba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:59:11.325827 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8eeb43-35ce-4f11-8bdb-c4be1f4bcc72 service placement] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:59:11.325827 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8eeb43-35ce-4f11-8bdb-c4be1f4bcc72 service placement] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:59:11.325827 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8eeb43-35ce-4f11-8bdb-c4be1f4bcc72 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:59:11.325827 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9e8eeb43-35ce-4f11-8bdb-c4be1f4bcc72 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:59:11.336621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e8eeb43-35ce-4f11-8bdb-c4be1f4bcc72 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:59:11.338183 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e8eeb43-35ce-4f11-8bdb-c4be1f4bcc72 service placement] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:59:11.348703 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1410/2827] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 09:59:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 09:59:11.403384 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c48f1c1-a2bb-401c-b7e7-bb1d34a81361 service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:59:11.404848 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-5c48f1c1-a2bb-401c-b7e7-bb1d34a81361 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:59:11.417446 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c48f1c1-a2bb-401c-b7e7-bb1d34a81361 service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:59:11.422205 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7adc14e8-cfc1-4219-b9ff-ca2fe709bca9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:59:11.422851 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adc14e8-cfc1-4219-b9ff-ca2fe709bca9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:59:11.423128 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adc14e8-cfc1-4219-b9ff-ca2fe709bca9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:59:11.423397 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adc14e8-cfc1-4219-b9ff-ca2fe709bca9 admin admin] PATH_INFO: `/v3/users/c1df4fe7054840feac0a648ce82eea1a` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:59:11.424531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7adc14e8-cfc1-4219-b9ff-ca2fe709bca9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c1df4fe7054840feac0a648ce82eea1a)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:59:11.426451 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7adc14e8-cfc1-4219-b9ff-ca2fe709bca9 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:59:11.535737 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7adc14e8-cfc1-4219-b9ff-ca2fe709bca9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c1df4fe7054840feac0a648ce82eea1a'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:59:11.547771 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7adc14e8-cfc1-4219-b9ff-ca2fe709bca9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c1df4fe7054840feac0a648ce82eea1a'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:59:11.548441 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7adc14e8-cfc1-4219-b9ff-ca2fe709bca9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c1df4fe7054840feac0a648ce82eea1a'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:59:11.585070 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7adc14e8-cfc1-4219-b9ff-ca2fe709bca9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c1df4fe7054840feac0a648ce82eea1a'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:59:11.591914 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-7adc14e8-cfc1-4219-b9ff-ca2fe709bca9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c1df4fe7054840feac0a648ce82eea1a'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:59:11.598379 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1418/2828] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:59:11 2022] DELETE /identity/v3/users/c1df4fe7054840feac0a648ce82eea1a => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:59:11.849603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8eeb43-35ce-4f11-8bdb-c4be1f4bcc72 service placement] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:59:11.861106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8eeb43-35ce-4f11-8bdb-c4be1f4bcc72 service placement] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:59:11.919799 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dcc824-f357-4e56-a2f7-737b2c7411a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:59:11.920448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dcc824-f357-4e56-a2f7-737b2c7411a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:59:11.920726 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dcc824-f357-4e56-a2f7-737b2c7411a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:59:11.921003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dcc824-f357-4e56-a2f7-737b2c7411a0 admin admin] PATH_INFO: `/v3/projects/f0d0c96bbb304882b1f2459361551a93` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:59:11.922128 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0dcc824-f357-4e56-a2f7-737b2c7411a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f0d0c96bbb304882b1f2459361551a93)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:59:11.924615 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0dcc824-f357-4e56-a2f7-737b2c7411a0 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:59:12.032242 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c0dcc824-f357-4e56-a2f7-737b2c7411a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f0d0c96bbb304882b1f2459361551a93', 'request_id': 'req-c0dcc824-f357-4e56-a2f7-737b2c7411a0'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:59:12.037991 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c0dcc824-f357-4e56-a2f7-737b2c7411a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f0d0c96bbb304882b1f2459361551a93', 'request_id': 'req-c0dcc824-f357-4e56-a2f7-737b2c7411a0'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:59:12.044253 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c0dcc824-f357-4e56-a2f7-737b2c7411a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f0d0c96bbb304882b1f2459361551a93', 'request_id': 'req-c0dcc824-f357-4e56-a2f7-737b2c7411a0'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:59:12.105593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c0dcc824-f357-4e56-a2f7-737b2c7411a0 admin admin] The token cache is being invalidate because project f0d0c96bbb304882b1f2459361551a93 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:59:12.108467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c0dcc824-f357-4e56-a2f7-737b2c7411a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:59:12.108467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c0dcc824-f357-4e56-a2f7-737b2c7411a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:59:12.108467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1411/2829] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:59:11 2022] DELETE /identity/v3/projects/f0d0c96bbb304882b1f2459361551a93 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:59:12.112582 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7adc14e8-cfc1-4219-b9ff-ca2fe709bca9 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:59:12.114078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7adc14e8-cfc1-4219-b9ff-ca2fe709bca9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:59:12.127747 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7adc14e8-cfc1-4219-b9ff-ca2fe709bca9 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:59:12.188809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295b70c5-c585-4479-b274-b66467835f56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:59:12.189472 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295b70c5-c585-4479-b274-b66467835f56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:59:12.190331 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295b70c5-c585-4479-b274-b66467835f56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:59:12.190641 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295b70c5-c585-4479-b274-b66467835f56 admin admin] PATH_INFO: `/v3/users/8a29648893a64ebb955d00833bc398e9` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:59:12.191797 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295b70c5-c585-4479-b274-b66467835f56 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8a29648893a64ebb955d00833bc398e9)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:59:12.196497 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295b70c5-c585-4479-b274-b66467835f56 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:59:12.360611 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-295b70c5-c585-4479-b274-b66467835f56 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8a29648893a64ebb955d00833bc398e9'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:59:12.372267 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=81,keystone:DELETE=18,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:59:12.373790 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-295b70c5-c585-4479-b274-b66467835f56 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8a29648893a64ebb955d00833bc398e9'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:59:12.374607 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-295b70c5-c585-4479-b274-b66467835f56 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8a29648893a64ebb955d00833bc398e9'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:59:12.454488 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-295b70c5-c585-4479-b274-b66467835f56 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8a29648893a64ebb955d00833bc398e9'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:59:12.461802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-295b70c5-c585-4479-b274-b66467835f56 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8a29648893a64ebb955d00833bc398e9'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:59:12.467790 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1419/2830] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 09:59:12 2022] DELETE /identity/v3/users/8a29648893a64ebb955d00833bc398e9 => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:59:12.680074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dcc824-f357-4e56-a2f7-737b2c7411a0 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 09:59:12.680552 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c0dcc824-f357-4e56-a2f7-737b2c7411a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 09:59:12.696565 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dcc824-f357-4e56-a2f7-737b2c7411a0 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 09:59:12.719714 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:59:12.759662 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aeffe95-4dde-4972-8561-0e4267a6f5ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 09:59:12.760303 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aeffe95-4dde-4972-8561-0e4267a6f5ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 09:59:12.760582 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aeffe95-4dde-4972-8561-0e4267a6f5ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 09:59:12.760860 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aeffe95-4dde-4972-8561-0e4267a6f5ae admin admin] PATH_INFO: `/v3/projects/8003aa7a2df842bb92467d64e17bab8d` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 09:59:12.762148 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aeffe95-4dde-4972-8561-0e4267a6f5ae admin admin] RBAC: Authorizing `identity:delete_project(project_id=8003aa7a2df842bb92467d64e17bab8d)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 09:59:12.764417 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aeffe95-4dde-4972-8561-0e4267a6f5ae admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 09:59:12.865365 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2aeffe95-4dde-4972-8561-0e4267a6f5ae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8003aa7a2df842bb92467d64e17bab8d', 'request_id': 'req-2aeffe95-4dde-4972-8561-0e4267a6f5ae'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:59:12.870788 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2aeffe95-4dde-4972-8561-0e4267a6f5ae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8003aa7a2df842bb92467d64e17bab8d', 'request_id': 'req-2aeffe95-4dde-4972-8561-0e4267a6f5ae'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:59:12.876800 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2aeffe95-4dde-4972-8561-0e4267a6f5ae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8003aa7a2df842bb92467d64e17bab8d', 'request_id': 'req-2aeffe95-4dde-4972-8561-0e4267a6f5ae'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:59:12.962526 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2aeffe95-4dde-4972-8561-0e4267a6f5ae admin admin] The token cache is being invalidate because project 8003aa7a2df842bb92467d64e17bab8d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 09:59:12.962773 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2aeffe95-4dde-4972-8561-0e4267a6f5ae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:59:12.963412 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2aeffe95-4dde-4972-8561-0e4267a6f5ae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 09:59:12.964902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1412/2831] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 09:59:12 2022] DELETE /identity/v3/projects/8003aa7a2df842bb92467d64e17bab8d => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 09:59:22.499923 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:DELETE=2,keystone:SELECT=7,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 09:59:22.959235 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=42,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:00:09.584619 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295b70c5-c585-4479-b274-b66467835f56 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:00:09.586061 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-295b70c5-c585-4479-b274-b66467835f56 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:00:09.654767 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295b70c5-c585-4479-b274-b66467835f56 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:00:09.712125 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227f670f-9f4d-4590-b9d7-fc746936c5e1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '49a3011c103b4e48904f876f81d27fba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:00:09.712715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227f670f-9f4d-4590-b9d7-fc746936c5e1 service placement] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:00:09.712982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227f670f-9f4d-4590-b9d7-fc746936c5e1 service placement] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:00:09.713246 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227f670f-9f4d-4590-b9d7-fc746936c5e1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:00:09.714969 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-227f670f-9f4d-4590-b9d7-fc746936c5e1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:00:09.726411 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227f670f-9f4d-4590-b9d7-fc746936c5e1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:00:09.728031 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227f670f-9f4d-4590-b9d7-fc746936c5e1 service placement] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:00:09.737664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1420/2832] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 10:00:09 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 10:00:19.733480 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=58 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:00:34.836642 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907c099a-bbe0-4391-affd-e78978e63169 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:00:34.839146 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907c099a-bbe0-4391-affd-e78978e63169 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:00:34.839146 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907c099a-bbe0-4391-affd-e78978e63169 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:00:34.856006 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-907c099a-bbe0-4391-affd-e78978e63169 None None] MFA Rules not processed for user `ee46fc4dd13143bfb409fb5033d9a4d2`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:00:34.857205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-907c099a-bbe0-4391-affd-e78978e63169 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:00:34.910688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1413/2833] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:00:34 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:00:44.904031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=25 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:01:25.942454 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227f670f-9f4d-4590-b9d7-fc746936c5e1 service placement] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:01:25.943924 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-227f670f-9f4d-4590-b9d7-fc746936c5e1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:01:26.001228 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227f670f-9f4d-4590-b9d7-fc746936c5e1 service placement] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:01:26.004918 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea54ded-5181-4a4a-9e72-ec4c1c0004a5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:01:26.005515 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea54ded-5181-4a4a-9e72-ec4c1c0004a5 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:01:26.005794 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea54ded-5181-4a4a-9e72-ec4c1c0004a5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:01:26.006062 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea54ded-5181-4a4a-9e72-ec4c1c0004a5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:01:26.008037 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-1ea54ded-5181-4a4a-9e72-ec4c1c0004a5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:01:26.078376 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea54ded-5181-4a4a-9e72-ec4c1c0004a5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:01:26.080109 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea54ded-5181-4a4a-9e72-ec4c1c0004a5 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:01:26.091517 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1421/2834] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 10:01:25 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 10:01:36.017155 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907c099a-bbe0-4391-affd-e78978e63169 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:01:36.018626 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-907c099a-bbe0-4391-affd-e78978e63169 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:01:36.030298 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907c099a-bbe0-4391-affd-e78978e63169 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:01:36.033799 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941a866c-60ef-45ce-a20b-596fc8fe1d85 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:01:36.034383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941a866c-60ef-45ce-a20b-596fc8fe1d85 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:01:36.034690 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941a866c-60ef-45ce-a20b-596fc8fe1d85 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:01:36.034963 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941a866c-60ef-45ce-a20b-596fc8fe1d85 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:01:36.036602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-941a866c-60ef-45ce-a20b-596fc8fe1d85 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:01:36.049755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941a866c-60ef-45ce-a20b-596fc8fe1d85 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:01:36.051205 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941a866c-60ef-45ce-a20b-596fc8fe1d85 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:01:36.062321 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1414/2835] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 10:01:36 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 10:01:36.086462 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=54 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:01:46.057093 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=6 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:01.458741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea54ded-5181-4a4a-9e72-ec4c1c0004a5 service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:01.460285 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-1ea54ded-5181-4a4a-9e72-ec4c1c0004a5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:01.525714 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea54ded-5181-4a4a-9e72-ec4c1c0004a5 service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:01.531081 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1b9b80-7055-4f0b-b5d1-f4ff127a8c6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:01.531847 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1b9b80-7055-4f0b-b5d1-f4ff127a8c6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:01.532121 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1b9b80-7055-4f0b-b5d1-f4ff127a8c6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:01.532391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1b9b80-7055-4f0b-b5d1-f4ff127a8c6a admin admin] PATH_INFO: `/v3/users/ee46fc4dd13143bfb409fb5033d9a4d2` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:01.533510 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1b9b80-7055-4f0b-b5d1-f4ff127a8c6a admin admin] RBAC: Authorizing `identity:delete_user(user_id=ee46fc4dd13143bfb409fb5033d9a4d2)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:01.535493 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1b9b80-7055-4f0b-b5d1-f4ff127a8c6a admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:01.698001 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-cf1b9b80-7055-4f0b-b5d1-f4ff127a8c6a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ee46fc4dd13143bfb409fb5033d9a4d2'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:01.710385 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-cf1b9b80-7055-4f0b-b5d1-f4ff127a8c6a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ee46fc4dd13143bfb409fb5033d9a4d2'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:01.711098 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-cf1b9b80-7055-4f0b-b5d1-f4ff127a8c6a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ee46fc4dd13143bfb409fb5033d9a4d2'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:01.764491 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-cf1b9b80-7055-4f0b-b5d1-f4ff127a8c6a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ee46fc4dd13143bfb409fb5033d9a4d2'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:01.771802 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-cf1b9b80-7055-4f0b-b5d1-f4ff127a8c6a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ee46fc4dd13143bfb409fb5033d9a4d2'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:01.777660 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1422/2836] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 10:02:01 2022] DELETE /identity/v3/users/ee46fc4dd13143bfb409fb5033d9a4d2 => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:02.005460 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941a866c-60ef-45ce-a20b-596fc8fe1d85 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:02.016484 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941a866c-60ef-45ce-a20b-596fc8fe1d85 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:02.073453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4dfdfe-c77a-45b5-8bee-88f943fbfe93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:02.074084 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4dfdfe-c77a-45b5-8bee-88f943fbfe93 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:02.074364 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4dfdfe-c77a-45b5-8bee-88f943fbfe93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:02.074662 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4dfdfe-c77a-45b5-8bee-88f943fbfe93 admin admin] PATH_INFO: `/v3/projects/85eef8fd790146cfa7da6214c13ca262` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:02.076264 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4dfdfe-c77a-45b5-8bee-88f943fbfe93 admin admin] RBAC: Authorizing `identity:delete_project(project_id=85eef8fd790146cfa7da6214c13ca262)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:02.078156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4dfdfe-c77a-45b5-8bee-88f943fbfe93 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:02.181300 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ac4dfdfe-c77a-45b5-8bee-88f943fbfe93 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '85eef8fd790146cfa7da6214c13ca262', 'request_id': 'req-ac4dfdfe-c77a-45b5-8bee-88f943fbfe93'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:02.186895 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ac4dfdfe-c77a-45b5-8bee-88f943fbfe93 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '85eef8fd790146cfa7da6214c13ca262', 'request_id': 'req-ac4dfdfe-c77a-45b5-8bee-88f943fbfe93'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:02.193004 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ac4dfdfe-c77a-45b5-8bee-88f943fbfe93 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '85eef8fd790146cfa7da6214c13ca262', 'request_id': 'req-ac4dfdfe-c77a-45b5-8bee-88f943fbfe93'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:02.347587 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ac4dfdfe-c77a-45b5-8bee-88f943fbfe93 admin admin] The token cache is being invalidate because project 85eef8fd790146cfa7da6214c13ca262 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 10:02:02.385612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ac4dfdfe-c77a-45b5-8bee-88f943fbfe93 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:02.385612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ac4dfdfe-c77a-45b5-8bee-88f943fbfe93 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:02.385612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1415/2837] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 10:02:02 2022] DELETE /identity/v3/projects/85eef8fd790146cfa7da6214c13ca262 => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:02.385950 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3d4886-a8b1-4058-b92f-2f673d4df663 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:02.385950 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3d4886-a8b1-4058-b92f-2f673d4df663 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:02.385950 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3d4886-a8b1-4058-b92f-2f673d4df663 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:02.385950 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-0f3d4886-a8b1-4058-b92f-2f673d4df663 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:02.385950 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-0f3d4886-a8b1-4058-b92f-2f673d4df663 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:02.433901 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1423/2838] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:02:02.442237 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026fc9e6-c638-48d6-8792-521047a31ff4 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:02.442527 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026fc9e6-c638-48d6-8792-521047a31ff4 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:02.442836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026fc9e6-c638-48d6-8792-521047a31ff4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:02.462724 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-026fc9e6-c638-48d6-8792-521047a31ff4 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:02.463930 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-026fc9e6-c638-48d6-8792-521047a31ff4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:02.471776 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1416/2839] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:02:02.475716 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3d4886-a8b1-4058-b92f-2f673d4df663 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:02.485531 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3d4886-a8b1-4058-b92f-2f673d4df663 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:02.490233 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9624b0f-2e04-475e-843d-2ed5e7fef40f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:02.490869 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9624b0f-2e04-475e-843d-2ed5e7fef40f admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:02.491152 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9624b0f-2e04-475e-843d-2ed5e7fef40f admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:02.491456 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9624b0f-2e04-475e-843d-2ed5e7fef40f admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:02.492092 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9624b0f-2e04-475e-843d-2ed5e7fef40f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 10:02:02.492424 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9624b0f-2e04-475e-843d-2ed5e7fef40f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:02.494028 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9624b0f-2e04-475e-843d-2ed5e7fef40f admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:02.505974 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1424/2840] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 10:02:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 10:02:02.510421 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026fc9e6-c638-48d6-8792-521047a31ff4 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:02.520273 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026fc9e6-c638-48d6-8792-521047a31ff4 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:02.524962 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0301fa7a-35e9-4045-b8f8-2cc5305b2340 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:02.525554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0301fa7a-35e9-4045-b8f8-2cc5305b2340 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:02.525836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0301fa7a-35e9-4045-b8f8-2cc5305b2340 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:02.526109 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0301fa7a-35e9-4045-b8f8-2cc5305b2340 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:02.526775 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0301fa7a-35e9-4045-b8f8-2cc5305b2340 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:02.528717 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0301fa7a-35e9-4045-b8f8-2cc5305b2340 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:02.605295 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1417/2841] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 10:02:02 2022] POST /identity/v3/projects => generated 375 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 10:02:02.609207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9624b0f-2e04-475e-843d-2ed5e7fef40f admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:02.619254 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9624b0f-2e04-475e-843d-2ed5e7fef40f admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:02.624285 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:02.675604 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c6e146-8662-4832-939f-cff92f916e4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:02.676205 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c6e146-8662-4832-939f-cff92f916e4d admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:02.676486 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c6e146-8662-4832-939f-cff92f916e4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:02.676762 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c6e146-8662-4832-939f-cff92f916e4d admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:02.677420 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c6e146-8662-4832-939f-cff92f916e4d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:02.680162 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c6e146-8662-4832-939f-cff92f916e4d admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:02.773579 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1425/2842] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 10:02:02 2022] POST /identity/v3/users => generated 288 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 10:02:02.775141 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0301fa7a-35e9-4045-b8f8-2cc5305b2340 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:02.786036 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0301fa7a-35e9-4045-b8f8-2cc5305b2340 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:02.790681 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f544a96-aa60-4a2c-9787-32e31c331a13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:02.791335 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f544a96-aa60-4a2c-9787-32e31c331a13 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:02.791612 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f544a96-aa60-4a2c-9787-32e31c331a13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:02.791888 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f544a96-aa60-4a2c-9787-32e31c331a13 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:02.792532 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f544a96-aa60-4a2c-9787-32e31c331a13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:02.794641 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f544a96-aa60-4a2c-9787-32e31c331a13 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:02.805259 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1418/2843] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 10:02:02 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 10:02:02.809602 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c6e146-8662-4832-939f-cff92f916e4d admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:02.821026 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c6e146-8662-4832-939f-cff92f916e4d admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:02.829911 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce37134c-0ca2-4d2e-bae0-767a6a431a7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:02.830555 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce37134c-0ca2-4d2e-bae0-767a6a431a7d admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:02.830830 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce37134c-0ca2-4d2e-bae0-767a6a431a7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:02.831101 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce37134c-0ca2-4d2e-bae0-767a6a431a7d admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:02.831741 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce37134c-0ca2-4d2e-bae0-767a6a431a7d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:02.833340 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce37134c-0ca2-4d2e-bae0-767a6a431a7d admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:02.845041 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1426/2844] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 10:02:02 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 10:02:02.848710 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f544a96-aa60-4a2c-9787-32e31c331a13 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:02.859415 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f544a96-aa60-4a2c-9787-32e31c331a13 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:02.863965 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41b9df6-1e05-46dd-a5cd-fc77187f40d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:02.864607 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41b9df6-1e05-46dd-a5cd-fc77187f40d0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:02.864889 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41b9df6-1e05-46dd-a5cd-fc77187f40d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:02.865191 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41b9df6-1e05-46dd-a5cd-fc77187f40d0 admin admin] PATH_INFO: `/v3/projects/079de817fc624bd4b00f344c664d951c/users/71898ee085d645bb959bef0e6494561d/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:02.880047 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41b9df6-1e05-46dd-a5cd-fc77187f40d0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=079de817fc624bd4b00f344c664d951c, user_id=71898ee085d645bb959bef0e6494561d, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:02.883035 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41b9df6-1e05-46dd-a5cd-fc77187f40d0 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:02.975859 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1419/2845] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 10:02:02 2022] PUT /identity/v3/projects/079de817fc624bd4b00f344c664d951c/users/71898ee085d645bb959bef0e6494561d/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:02.987108 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3abaf87-88d5-44eb-a692-dcd1bb5409b9 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:02.988481 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3abaf87-88d5-44eb-a692-dcd1bb5409b9 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:02.988481 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3abaf87-88d5-44eb-a692-dcd1bb5409b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:03.016896 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-f3abaf87-88d5-44eb-a692-dcd1bb5409b9 None None] MFA Rules not processed for user `71898ee085d645bb959bef0e6494561d`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:03.018124 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-f3abaf87-88d5-44eb-a692-dcd1bb5409b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:03.080988 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1427/2846] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:02 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:02:03.092199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41b9df6-1e05-46dd-a5cd-fc77187f40d0 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:03.093955 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e41b9df6-1e05-46dd-a5cd-fc77187f40d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:03.125424 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:03.167690 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41b9df6-1e05-46dd-a5cd-fc77187f40d0 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:03.174256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af5bd12-9e83-4168-8f4d-2c5aa44351b6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ff68fb87700423ab53c90520a6bd3b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:03.174896 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af5bd12-9e83-4168-8f4d-2c5aa44351b6 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:03.175178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af5bd12-9e83-4168-8f4d-2c5aa44351b6 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:03.175453 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af5bd12-9e83-4168-8f4d-2c5aa44351b6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:03.183443 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af5bd12-9e83-4168-8f4d-2c5aa44351b6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:03.185672 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af5bd12-9e83-4168-8f4d-2c5aa44351b6 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:03.195179 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1420/2847] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 10:02:03 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 10:02:03.397425 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3abaf87-88d5-44eb-a692-dcd1bb5409b9 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:03.457540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3abaf87-88d5-44eb-a692-dcd1bb5409b9 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:03.461952 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd37c02-376d-4b66-9850-8afde49e53f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:03.462564 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd37c02-376d-4b66-9850-8afde49e53f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:03.462870 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd37c02-376d-4b66-9850-8afde49e53f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:03.463140 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd37c02-376d-4b66-9850-8afde49e53f1 admin admin] PATH_INFO: `/v3/users/71898ee085d645bb959bef0e6494561d` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:03.464227 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd37c02-376d-4b66-9850-8afde49e53f1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=71898ee085d645bb959bef0e6494561d)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:03.466661 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd37c02-376d-4b66-9850-8afde49e53f1 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:03.481402 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:03.635716 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-edd37c02-376d-4b66-9850-8afde49e53f1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '71898ee085d645bb959bef0e6494561d'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:03.648186 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-edd37c02-376d-4b66-9850-8afde49e53f1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '71898ee085d645bb959bef0e6494561d'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:03.648850 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-edd37c02-376d-4b66-9850-8afde49e53f1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '71898ee085d645bb959bef0e6494561d'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:03.689519 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-edd37c02-376d-4b66-9850-8afde49e53f1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '71898ee085d645bb959bef0e6494561d'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:03.696729 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-edd37c02-376d-4b66-9850-8afde49e53f1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '71898ee085d645bb959bef0e6494561d'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:03.703455 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1428/2848] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 10:02:03 2022] DELETE /identity/v3/users/71898ee085d645bb959bef0e6494561d => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:03.707941 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c2ed6a-5e19-4cdf-abb5-3802c9ef0e93 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:03.709245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c2ed6a-5e19-4cdf-abb5-3802c9ef0e93 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:03.709245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c2ed6a-5e19-4cdf-abb5-3802c9ef0e93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:03.727766 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d7c2ed6a-5e19-4cdf-abb5-3802c9ef0e93 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:03.728959 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d7c2ed6a-5e19-4cdf-abb5-3802c9ef0e93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:03.785168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1421/2849] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:02:03.794703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd37c02-376d-4b66-9850-8afde49e53f1 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:03.796120 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-edd37c02-376d-4b66-9850-8afde49e53f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:03.853607 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd37c02-376d-4b66-9850-8afde49e53f1 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:03.857947 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88ae88e-4a3e-47b3-865f-682fe5b75d63 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:03.858553 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88ae88e-4a3e-47b3-865f-682fe5b75d63 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:03.858867 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88ae88e-4a3e-47b3-865f-682fe5b75d63 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:03.859140 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88ae88e-4a3e-47b3-865f-682fe5b75d63 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:03.868257 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88ae88e-4a3e-47b3-865f-682fe5b75d63 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:03.869760 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88ae88e-4a3e-47b3-865f-682fe5b75d63 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:03.879209 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1429/2850] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 10:02:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 10:02:04.251904 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c2ed6a-5e19-4cdf-abb5-3802c9ef0e93 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:04.253603 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d7c2ed6a-5e19-4cdf-abb5-3802c9ef0e93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:04.266496 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c2ed6a-5e19-4cdf-abb5-3802c9ef0e93 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:04.271271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336de373-1a71-4dbe-bce7-49860a763d2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:04.272180 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336de373-1a71-4dbe-bce7-49860a763d2d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:04.272456 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336de373-1a71-4dbe-bce7-49860a763d2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:04.272750 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336de373-1a71-4dbe-bce7-49860a763d2d admin admin] PATH_INFO: `/v3/projects/079de817fc624bd4b00f344c664d951c` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:04.273903 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336de373-1a71-4dbe-bce7-49860a763d2d admin admin] RBAC: Authorizing `identity:delete_project(project_id=079de817fc624bd4b00f344c664d951c)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:04.275908 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336de373-1a71-4dbe-bce7-49860a763d2d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:04.393455 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-336de373-1a71-4dbe-bce7-49860a763d2d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '079de817fc624bd4b00f344c664d951c', 'request_id': 'req-336de373-1a71-4dbe-bce7-49860a763d2d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:04.399009 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-336de373-1a71-4dbe-bce7-49860a763d2d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '079de817fc624bd4b00f344c664d951c', 'request_id': 'req-336de373-1a71-4dbe-bce7-49860a763d2d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:04.405271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-336de373-1a71-4dbe-bce7-49860a763d2d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '079de817fc624bd4b00f344c664d951c', 'request_id': 'req-336de373-1a71-4dbe-bce7-49860a763d2d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:04.466655 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-336de373-1a71-4dbe-bce7-49860a763d2d admin admin] The token cache is being invalidate because project 079de817fc624bd4b00f344c664d951c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 10:02:04.466869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-336de373-1a71-4dbe-bce7-49860a763d2d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:04.467481 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-336de373-1a71-4dbe-bce7-49860a763d2d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:04.468972 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1422/2851] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 10:02:04 2022] DELETE /identity/v3/projects/079de817fc624bd4b00f344c664d951c => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:04.474555 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ca5310-8172-4151-bde1-b1a3c89e20fd None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:04.474877 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ca5310-8172-4151-bde1-b1a3c89e20fd None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:04.475155 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ca5310-8172-4151-bde1-b1a3c89e20fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:04.495111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-95ca5310-8172-4151-bde1-b1a3c89e20fd None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:04.501608 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-95ca5310-8172-4151-bde1-b1a3c89e20fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:04.557311 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1430/2852] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:02:04.566682 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993dc955-147a-435b-979c-9e33b7df4a8a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:04.567000 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993dc955-147a-435b-979c-9e33b7df4a8a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:04.567278 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993dc955-147a-435b-979c-9e33b7df4a8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:04.587963 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-993dc955-147a-435b-979c-9e33b7df4a8a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:04.589161 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-993dc955-147a-435b-979c-9e33b7df4a8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:04.597792 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1423/2853] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:02:04.602009 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ca5310-8172-4151-bde1-b1a3c89e20fd None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:04.611970 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ca5310-8172-4151-bde1-b1a3c89e20fd None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:04.616744 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de68f7d-ee4e-4e9e-b3f6-4a6b0259fedc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:04.617345 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de68f7d-ee4e-4e9e-b3f6-4a6b0259fedc admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:04.617629 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de68f7d-ee4e-4e9e-b3f6-4a6b0259fedc admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:04.617918 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de68f7d-ee4e-4e9e-b3f6-4a6b0259fedc admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:04.618614 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de68f7d-ee4e-4e9e-b3f6-4a6b0259fedc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 10:02:04.618950 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de68f7d-ee4e-4e9e-b3f6-4a6b0259fedc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:04.620551 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de68f7d-ee4e-4e9e-b3f6-4a6b0259fedc admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:04.632881 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1431/2854] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 10:02:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 10:02:04.637215 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993dc955-147a-435b-979c-9e33b7df4a8a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:04.646223 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993dc955-147a-435b-979c-9e33b7df4a8a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:04.650754 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a6365e-79cf-4a26-81ee-73adc154a990 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:04.651337 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a6365e-79cf-4a26-81ee-73adc154a990 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:04.651611 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a6365e-79cf-4a26-81ee-73adc154a990 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:04.651915 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a6365e-79cf-4a26-81ee-73adc154a990 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:04.652560 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a6365e-79cf-4a26-81ee-73adc154a990 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:04.654425 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a6365e-79cf-4a26-81ee-73adc154a990 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:04.747599 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1424/2855] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 10:02:04 2022] POST /identity/v3/projects => generated 381 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 10:02:04.751651 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de68f7d-ee4e-4e9e-b3f6-4a6b0259fedc admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:04.762230 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de68f7d-ee4e-4e9e-b3f6-4a6b0259fedc admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:04.782815 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:04.819132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c26ef7e-9b6d-492a-9565-f5ef1bf9d268 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:04.819727 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c26ef7e-9b6d-492a-9565-f5ef1bf9d268 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:04.820002 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c26ef7e-9b6d-492a-9565-f5ef1bf9d268 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:04.820271 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c26ef7e-9b6d-492a-9565-f5ef1bf9d268 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:04.820895 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c26ef7e-9b6d-492a-9565-f5ef1bf9d268 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:04.822939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c26ef7e-9b6d-492a-9565-f5ef1bf9d268 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:04.895664 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1432/2856] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 10:02:04 2022] POST /identity/v3/users => generated 291 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 10:02:04.899742 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a6365e-79cf-4a26-81ee-73adc154a990 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:04.910632 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a6365e-79cf-4a26-81ee-73adc154a990 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:04.915420 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972253f9-d6bc-4aa0-a0c1-5960a5373e70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:04.916018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972253f9-d6bc-4aa0-a0c1-5960a5373e70 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:04.916298 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972253f9-d6bc-4aa0-a0c1-5960a5373e70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:04.916573 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972253f9-d6bc-4aa0-a0c1-5960a5373e70 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:04.917381 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972253f9-d6bc-4aa0-a0c1-5960a5373e70 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:04.919592 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972253f9-d6bc-4aa0-a0c1-5960a5373e70 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:04.920826 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:04.930516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1425/2857] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 10:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 10:02:04.934513 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c26ef7e-9b6d-492a-9565-f5ef1bf9d268 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:04.944473 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c26ef7e-9b6d-492a-9565-f5ef1bf9d268 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:04.949111 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f782f78-47a6-435f-8f86-020d110e90fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:04.949698 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f782f78-47a6-435f-8f86-020d110e90fd admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:04.949976 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f782f78-47a6-435f-8f86-020d110e90fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:04.950246 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f782f78-47a6-435f-8f86-020d110e90fd admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:04.950940 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f782f78-47a6-435f-8f86-020d110e90fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:04.952534 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f782f78-47a6-435f-8f86-020d110e90fd admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:04.962251 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1433/2858] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 10:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 10:02:04.967144 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972253f9-d6bc-4aa0-a0c1-5960a5373e70 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:04.976880 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972253f9-d6bc-4aa0-a0c1-5960a5373e70 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:04.982470 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92e5336-c404-48a3-95b9-d20a3d67da5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:04.983204 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92e5336-c404-48a3-95b9-d20a3d67da5b admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:04.983814 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92e5336-c404-48a3-95b9-d20a3d67da5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:04.984140 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92e5336-c404-48a3-95b9-d20a3d67da5b admin admin] PATH_INFO: `/v3/projects/8e0743ef9beb45739ba705ffdce70162/users/9707c60ee4df4c4ca05e380aa0077e42/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:04.998781 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92e5336-c404-48a3-95b9-d20a3d67da5b admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e0743ef9beb45739ba705ffdce70162, user_id=9707c60ee4df4c4ca05e380aa0077e42, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:05.001404 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92e5336-c404-48a3-95b9-d20a3d67da5b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:05.055325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1426/2859] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 10:02:04 2022] PUT /identity/v3/projects/8e0743ef9beb45739ba705ffdce70162/users/9707c60ee4df4c4ca05e380aa0077e42/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:05.065555 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907c9dfd-88b4-45d5-a5e2-8dc4fd763d4c None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:05.065872 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907c9dfd-88b4-45d5-a5e2-8dc4fd763d4c None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:05.066172 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907c9dfd-88b4-45d5-a5e2-8dc4fd763d4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:05.094383 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-907c9dfd-88b4-45d5-a5e2-8dc4fd763d4c None None] MFA Rules not processed for user `9707c60ee4df4c4ca05e380aa0077e42`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:05.095615 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-907c9dfd-88b4-45d5-a5e2-8dc4fd763d4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:05.159178 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1434/2860] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:05 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:02:05.163625 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92e5336-c404-48a3-95b9-d20a3d67da5b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:05.224877 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92e5336-c404-48a3-95b9-d20a3d67da5b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:05.229470 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd11c6f-1333-4ede-b105-bc191330330b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:05.230078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd11c6f-1333-4ede-b105-bc191330330b admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:05.230515 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd11c6f-1333-4ede-b105-bc191330330b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:05.230827 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd11c6f-1333-4ede-b105-bc191330330b admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:05.231502 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd11c6f-1333-4ede-b105-bc191330330b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:05.233413 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd11c6f-1333-4ede-b105-bc191330330b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:05.296201 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1427/2861] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 10:02:05 2022] POST /identity/v3/projects => generated 381 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 10:02:05.300241 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907c9dfd-88b4-45d5-a5e2-8dc4fd763d4c None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:05.310956 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907c9dfd-88b4-45d5-a5e2-8dc4fd763d4c None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:05.367174 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e53935-f185-45c3-adb1-3753f6fec2ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:05.367782 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e53935-f185-45c3-adb1-3753f6fec2ba admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:05.368071 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e53935-f185-45c3-adb1-3753f6fec2ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:05.368382 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e53935-f185-45c3-adb1-3753f6fec2ba admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:05.369018 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e53935-f185-45c3-adb1-3753f6fec2ba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:05.371002 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e53935-f185-45c3-adb1-3753f6fec2ba admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:05.484112 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1435/2862] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 10:02:05 2022] POST /identity/v3/users => generated 297 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 30 10:02:05.488000 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd11c6f-1333-4ede-b105-bc191330330b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:05.498580 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd11c6f-1333-4ede-b105-bc191330330b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:05.503130 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b7b1b7-67d0-4c23-9a2e-a1731ac99c38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:05.503720 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b7b1b7-67d0-4c23-9a2e-a1731ac99c38 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:05.503995 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b7b1b7-67d0-4c23-9a2e-a1731ac99c38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:05.504265 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b7b1b7-67d0-4c23-9a2e-a1731ac99c38 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:05.504891 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b7b1b7-67d0-4c23-9a2e-a1731ac99c38 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:05.506516 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b7b1b7-67d0-4c23-9a2e-a1731ac99c38 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:05.517621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1428/2863] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 10:02:05 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 10:02:05.521578 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e53935-f185-45c3-adb1-3753f6fec2ba admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:05.531454 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e53935-f185-45c3-adb1-3753f6fec2ba admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:05.536122 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1312935b-5043-4a67-9f21-13114b66cccb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:05.536754 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1312935b-5043-4a67-9f21-13114b66cccb admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:05.537032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1312935b-5043-4a67-9f21-13114b66cccb admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:05.537304 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1312935b-5043-4a67-9f21-13114b66cccb admin admin] PATH_INFO: `/v3/projects/ef63e4186cc34e35a5244d9efbc38bf0/users/df1d838e357f4f46993bc7743b095c9e/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:05.551779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1312935b-5043-4a67-9f21-13114b66cccb admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef63e4186cc34e35a5244d9efbc38bf0, user_id=df1d838e357f4f46993bc7743b095c9e, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:05.554351 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1312935b-5043-4a67-9f21-13114b66cccb admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:05.645188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:05.650831 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1436/2864] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 10:02:05 2022] PUT /identity/v3/projects/ef63e4186cc34e35a5244d9efbc38bf0/users/df1d838e357f4f46993bc7743b095c9e/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:05.654829 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b7b1b7-67d0-4c23-9a2e-a1731ac99c38 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:05.664834 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b7b1b7-67d0-4c23-9a2e-a1731ac99c38 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:05.719795 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce85cdc4-4412-4e88-b5d7-40aeb12d1b9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:05.720396 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce85cdc4-4412-4e88-b5d7-40aeb12d1b9f admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:05.720676 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce85cdc4-4412-4e88-b5d7-40aeb12d1b9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:05.720949 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce85cdc4-4412-4e88-b5d7-40aeb12d1b9f admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:05.721582 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce85cdc4-4412-4e88-b5d7-40aeb12d1b9f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:05.723236 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce85cdc4-4412-4e88-b5d7-40aeb12d1b9f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:05.732980 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1429/2865] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 10:02:05 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 10:02:05.737029 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1312935b-5043-4a67-9f21-13114b66cccb admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:05.747867 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1312935b-5043-4a67-9f21-13114b66cccb admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:05.752559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca2bce7-05ed-4498-8b74-d05db5efca92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:05.753199 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca2bce7-05ed-4498-8b74-d05db5efca92 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:05.753481 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca2bce7-05ed-4498-8b74-d05db5efca92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:05.753755 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca2bce7-05ed-4498-8b74-d05db5efca92 admin admin] PATH_INFO: `/v3/projects/ef63e4186cc34e35a5244d9efbc38bf0/users/df1d838e357f4f46993bc7743b095c9e/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:05.755711 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca2bce7-05ed-4498-8b74-d05db5efca92 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef63e4186cc34e35a5244d9efbc38bf0, user_id=df1d838e357f4f46993bc7743b095c9e, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:05.758699 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca2bce7-05ed-4498-8b74-d05db5efca92 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:05.769412 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1437/2866] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 10:02:05 2022] PUT /identity/v3/projects/ef63e4186cc34e35a5244d9efbc38bf0/users/df1d838e357f4f46993bc7743b095c9e/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:05.779436 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6136d28-8c4b-4ad3-8f67-1fa059a5fddc None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:05.779732 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6136d28-8c4b-4ad3-8f67-1fa059a5fddc None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:05.780008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6136d28-8c4b-4ad3-8f67-1fa059a5fddc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:05.809304 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e6136d28-8c4b-4ad3-8f67-1fa059a5fddc None None] MFA Rules not processed for user `df1d838e357f4f46993bc7743b095c9e`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:05.809304 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e6136d28-8c4b-4ad3-8f67-1fa059a5fddc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:05.839333 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:05.877272 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1430/2867] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:05 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 10:02:05.888233 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca2bce7-05ed-4498-8b74-d05db5efca92 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:05.890064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-7ca2bce7-05ed-4498-8b74-d05db5efca92 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:05.950547 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca2bce7-05ed-4498-8b74-d05db5efca92 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:05.954268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969b48bf-0c50-42d2-b694-cd81b018ff59 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:05.954903 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969b48bf-0c50-42d2-b694-cd81b018ff59 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:05.955183 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969b48bf-0c50-42d2-b694-cd81b018ff59 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:05.955459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969b48bf-0c50-42d2-b694-cd81b018ff59 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:05.964290 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969b48bf-0c50-42d2-b694-cd81b018ff59 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:05.965757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969b48bf-0c50-42d2-b694-cd81b018ff59 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:05.975067 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1438/2868] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 10:02:05 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 10:02:06.076158 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6136d28-8c4b-4ad3-8f67-1fa059a5fddc None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:06.085809 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6136d28-8c4b-4ad3-8f67-1fa059a5fddc None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:06.089418 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258e7092-3da0-44b2-b3b2-cf1f2977ab17 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:06.090012 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258e7092-3da0-44b2-b3b2-cf1f2977ab17 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:06.090294 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258e7092-3da0-44b2-b3b2-cf1f2977ab17 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:06.090571 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258e7092-3da0-44b2-b3b2-cf1f2977ab17 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:06.147893 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258e7092-3da0-44b2-b3b2-cf1f2977ab17 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:06.149329 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258e7092-3da0-44b2-b3b2-cf1f2977ab17 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:06.158089 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1431/2869] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 10:02:06 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 10:02:09.113561 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969b48bf-0c50-42d2-b694-cd81b018ff59 service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:09.174263 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969b48bf-0c50-42d2-b694-cd81b018ff59 service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:09.178862 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cdc704-85a7-40af-9f12-0dff9a5e1f38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:09.179492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cdc704-85a7-40af-9f12-0dff9a5e1f38 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:09.179771 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cdc704-85a7-40af-9f12-0dff9a5e1f38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:09.180045 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cdc704-85a7-40af-9f12-0dff9a5e1f38 admin admin] PATH_INFO: `/v3/users/9707c60ee4df4c4ca05e380aa0077e42` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:09.181169 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cdc704-85a7-40af-9f12-0dff9a5e1f38 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9707c60ee4df4c4ca05e380aa0077e42)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:09.183130 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cdc704-85a7-40af-9f12-0dff9a5e1f38 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:09.314939 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-b9cdc704-85a7-40af-9f12-0dff9a5e1f38 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9707c60ee4df4c4ca05e380aa0077e42'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:09.326615 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-b9cdc704-85a7-40af-9f12-0dff9a5e1f38 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9707c60ee4df4c4ca05e380aa0077e42'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:09.327280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-b9cdc704-85a7-40af-9f12-0dff9a5e1f38 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9707c60ee4df4c4ca05e380aa0077e42'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:09.392078 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-b9cdc704-85a7-40af-9f12-0dff9a5e1f38 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9707c60ee4df4c4ca05e380aa0077e42'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:09.399064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-b9cdc704-85a7-40af-9f12-0dff9a5e1f38 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9707c60ee4df4c4ca05e380aa0077e42'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:09.405499 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1439/2870] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 10:02:09 2022] DELETE /identity/v3/users/9707c60ee4df4c4ca05e380aa0077e42 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:09.410018 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223df29c-b7b0-4a4b-a759-5236b4ee3a55 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:09.410500 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223df29c-b7b0-4a4b-a759-5236b4ee3a55 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:09.410644 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223df29c-b7b0-4a4b-a759-5236b4ee3a55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:09.429688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-223df29c-b7b0-4a4b-a759-5236b4ee3a55 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:09.430902 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-223df29c-b7b0-4a4b-a759-5236b4ee3a55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:09.487723 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1432/2871] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:02:09.497275 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cdc704-85a7-40af-9f12-0dff9a5e1f38 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:09.498706 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b9cdc704-85a7-40af-9f12-0dff9a5e1f38 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:09.508678 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:09.556228 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cdc704-85a7-40af-9f12-0dff9a5e1f38 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:09.559965 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e831443a-a235-4f2b-8252-6b9a768e7f75 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:09.560559 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e831443a-a235-4f2b-8252-6b9a768e7f75 service neutron] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:09.560839 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e831443a-a235-4f2b-8252-6b9a768e7f75 service neutron] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:09.561113 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e831443a-a235-4f2b-8252-6b9a768e7f75 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:09.570734 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e831443a-a235-4f2b-8252-6b9a768e7f75 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:09.572194 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e831443a-a235-4f2b-8252-6b9a768e7f75 service neutron] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:09.582223 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1440/2872] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 10:02:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 30 10:02:09.970852 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223df29c-b7b0-4a4b-a759-5236b4ee3a55 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:09.972280 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-223df29c-b7b0-4a4b-a759-5236b4ee3a55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:09.986467 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223df29c-b7b0-4a4b-a759-5236b4ee3a55 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:09.991702 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296dd0a5-e881-4d5e-a291-d2130dd51e5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:09.992494 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296dd0a5-e881-4d5e-a291-d2130dd51e5b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:09.992804 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296dd0a5-e881-4d5e-a291-d2130dd51e5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:09.993078 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296dd0a5-e881-4d5e-a291-d2130dd51e5b admin admin] PATH_INFO: `/v3/projects/8e0743ef9beb45739ba705ffdce70162` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:09.994210 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296dd0a5-e881-4d5e-a291-d2130dd51e5b admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e0743ef9beb45739ba705ffdce70162)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:09.996156 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296dd0a5-e881-4d5e-a291-d2130dd51e5b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:10.078649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-296dd0a5-e881-4d5e-a291-d2130dd51e5b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e0743ef9beb45739ba705ffdce70162', 'request_id': 'req-296dd0a5-e881-4d5e-a291-d2130dd51e5b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:10.084209 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-296dd0a5-e881-4d5e-a291-d2130dd51e5b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e0743ef9beb45739ba705ffdce70162', 'request_id': 'req-296dd0a5-e881-4d5e-a291-d2130dd51e5b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:10.086049 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:10.090458 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-296dd0a5-e881-4d5e-a291-d2130dd51e5b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e0743ef9beb45739ba705ffdce70162', 'request_id': 'req-296dd0a5-e881-4d5e-a291-d2130dd51e5b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:10.151884 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-296dd0a5-e881-4d5e-a291-d2130dd51e5b admin admin] The token cache is being invalidate because project 8e0743ef9beb45739ba705ffdce70162 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 10:02:10.152202 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-296dd0a5-e881-4d5e-a291-d2130dd51e5b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:10.152842 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-296dd0a5-e881-4d5e-a291-d2130dd51e5b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:10.154353 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1433/2873] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 10:02:09 2022] DELETE /identity/v3/projects/8e0743ef9beb45739ba705ffdce70162 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:10.158509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e831443a-a235-4f2b-8252-6b9a768e7f75 service neutron] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:10.168123 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e831443a-a235-4f2b-8252-6b9a768e7f75 service neutron] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:10.224365 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d303e83-043e-42b1-b0b1-8c4c897de2fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:10.224993 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d303e83-043e-42b1-b0b1-8c4c897de2fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:10.225270 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d303e83-043e-42b1-b0b1-8c4c897de2fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:10.225544 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d303e83-043e-42b1-b0b1-8c4c897de2fc admin admin] PATH_INFO: `/v3/users/df1d838e357f4f46993bc7743b095c9e` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:10.226674 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d303e83-043e-42b1-b0b1-8c4c897de2fc admin admin] RBAC: Authorizing `identity:delete_user(user_id=df1d838e357f4f46993bc7743b095c9e)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:10.228602 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d303e83-043e-42b1-b0b1-8c4c897de2fc admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:10.411453 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-9d303e83-043e-42b1-b0b1-8c4c897de2fc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'df1d838e357f4f46993bc7743b095c9e'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:10.423888 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-9d303e83-043e-42b1-b0b1-8c4c897de2fc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'df1d838e357f4f46993bc7743b095c9e'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:10.424534 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-9d303e83-043e-42b1-b0b1-8c4c897de2fc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'df1d838e357f4f46993bc7743b095c9e'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:10.477990 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-9d303e83-043e-42b1-b0b1-8c4c897de2fc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'df1d838e357f4f46993bc7743b095c9e'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:10.485314 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-9d303e83-043e-42b1-b0b1-8c4c897de2fc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'df1d838e357f4f46993bc7743b095c9e'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:10.491188 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1441/2874] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 10:02:10 2022] DELETE /identity/v3/users/df1d838e357f4f46993bc7743b095c9e => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:10.890903 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296dd0a5-e881-4d5e-a291-d2130dd51e5b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:10.892346 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-296dd0a5-e881-4d5e-a291-d2130dd51e5b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:10.906181 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296dd0a5-e881-4d5e-a291-d2130dd51e5b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:10.963783 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a6d22b-259a-49db-9253-94ecd1a5446e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:10.964410 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a6d22b-259a-49db-9253-94ecd1a5446e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:10.964686 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a6d22b-259a-49db-9253-94ecd1a5446e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:10.964957 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a6d22b-259a-49db-9253-94ecd1a5446e admin admin] PATH_INFO: `/v3/projects/ef63e4186cc34e35a5244d9efbc38bf0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:10.966622 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a6d22b-259a-49db-9253-94ecd1a5446e admin admin] RBAC: Authorizing `identity:delete_project(project_id=ef63e4186cc34e35a5244d9efbc38bf0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:10.968553 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a6d22b-259a-49db-9253-94ecd1a5446e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:11.061832 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a9a6d22b-259a-49db-9253-94ecd1a5446e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ef63e4186cc34e35a5244d9efbc38bf0', 'request_id': 'req-a9a6d22b-259a-49db-9253-94ecd1a5446e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:11.067196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a9a6d22b-259a-49db-9253-94ecd1a5446e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ef63e4186cc34e35a5244d9efbc38bf0', 'request_id': 'req-a9a6d22b-259a-49db-9253-94ecd1a5446e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:11.073304 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a9a6d22b-259a-49db-9253-94ecd1a5446e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ef63e4186cc34e35a5244d9efbc38bf0', 'request_id': 'req-a9a6d22b-259a-49db-9253-94ecd1a5446e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:11.175031 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a9a6d22b-259a-49db-9253-94ecd1a5446e admin admin] The token cache is being invalidate because project ef63e4186cc34e35a5244d9efbc38bf0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 10:02:11.175325 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a9a6d22b-259a-49db-9253-94ecd1a5446e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:11.175941 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-a9a6d22b-259a-49db-9253-94ecd1a5446e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:11.177478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1434/2875] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 10:02:10 2022] DELETE /identity/v3/projects/ef63e4186cc34e35a5244d9efbc38bf0 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:11.182712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99b2df2-d15f-4a9f-aaed-7c38885d7b92 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:11.183204 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99b2df2-d15f-4a9f-aaed-7c38885d7b92 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:11.183342 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99b2df2-d15f-4a9f-aaed-7c38885d7b92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:11.204393 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-d99b2df2-d15f-4a9f-aaed-7c38885d7b92 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:11.205631 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d99b2df2-d15f-4a9f-aaed-7c38885d7b92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:11.230315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:11.265767 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1442/2876] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:02:11.275879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50abcaa-cd74-496e-a251-22283177ce4a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:11.275879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50abcaa-cd74-496e-a251-22283177ce4a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:11.275879 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50abcaa-cd74-496e-a251-22283177ce4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:11.295489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c50abcaa-cd74-496e-a251-22283177ce4a None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:11.296708 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c50abcaa-cd74-496e-a251-22283177ce4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:11.304537 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1435/2877] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:02:11.310388 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99b2df2-d15f-4a9f-aaed-7c38885d7b92 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:11.318357 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99b2df2-d15f-4a9f-aaed-7c38885d7b92 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:11.322910 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8dfd53-18b0-4245-9a7a-86e6e2db5f23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:11.323499 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8dfd53-18b0-4245-9a7a-86e6e2db5f23 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:11.323776 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8dfd53-18b0-4245-9a7a-86e6e2db5f23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:11.324043 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8dfd53-18b0-4245-9a7a-86e6e2db5f23 admin admin] PATH_INFO: `/v3/domains` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:11.324672 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8dfd53-18b0-4245-9a7a-86e6e2db5f23 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81685) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 10:02:11.325503 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8dfd53-18b0-4245-9a7a-86e6e2db5f23 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:11.327120 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8dfd53-18b0-4245-9a7a-86e6e2db5f23 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:11.339448 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1443/2878] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 10:02:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 10:02:11.348070 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50abcaa-cd74-496e-a251-22283177ce4a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:11.357269 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50abcaa-cd74-496e-a251-22283177ce4a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:11.361830 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb07fb8c-85dc-4c98-ab56-fad732cf1eb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:11.362448 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb07fb8c-85dc-4c98-ab56-fad732cf1eb3 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:11.362759 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb07fb8c-85dc-4c98-ab56-fad732cf1eb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:11.363029 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb07fb8c-85dc-4c98-ab56-fad732cf1eb3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:11.363649 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb07fb8c-85dc-4c98-ab56-fad732cf1eb3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:11.365554 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb07fb8c-85dc-4c98-ab56-fad732cf1eb3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:11.444883 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1436/2879] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 10:02:11 2022] POST /identity/v3/projects => generated 377 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 10:02:11.449198 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8dfd53-18b0-4245-9a7a-86e6e2db5f23 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:11.460027 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8dfd53-18b0-4245-9a7a-86e6e2db5f23 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:11.516178 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fdbbf9-ae18-4980-ab97-c00d9decfbc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:11.516785 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fdbbf9-ae18-4980-ab97-c00d9decfbc5 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:11.517162 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fdbbf9-ae18-4980-ab97-c00d9decfbc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:11.517442 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fdbbf9-ae18-4980-ab97-c00d9decfbc5 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:11.518080 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fdbbf9-ae18-4980-ab97-c00d9decfbc5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:11.520047 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fdbbf9-ae18-4980-ab97-c00d9decfbc5 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:11.603796 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1444/2880] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 10:02:11 2022] POST /identity/v3/users => generated 289 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 10:02:11.607998 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb07fb8c-85dc-4c98-ab56-fad732cf1eb3 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:11.618394 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb07fb8c-85dc-4c98-ab56-fad732cf1eb3 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:11.623630 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d9fce3-12f2-43f0-bab0-7157a1f6e6f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:11.624222 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d9fce3-12f2-43f0-bab0-7157a1f6e6f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:11.624553 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d9fce3-12f2-43f0-bab0-7157a1f6e6f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:11.624779 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d9fce3-12f2-43f0-bab0-7157a1f6e6f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:11.625423 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d9fce3-12f2-43f0-bab0-7157a1f6e6f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:11.627097 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d9fce3-12f2-43f0-bab0-7157a1f6e6f8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:11.638319 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1437/2881] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 10:02:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 10:02:11.642554 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fdbbf9-ae18-4980-ab97-c00d9decfbc5 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:11.658733 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fdbbf9-ae18-4980-ab97-c00d9decfbc5 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:11.663398 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928da710-1522-4913-bc2f-068793392075 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:11.663989 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928da710-1522-4913-bc2f-068793392075 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:11.664272 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928da710-1522-4913-bc2f-068793392075 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:11.664544 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928da710-1522-4913-bc2f-068793392075 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:11.665202 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928da710-1522-4913-bc2f-068793392075 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:11.666805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928da710-1522-4913-bc2f-068793392075 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:11.677617 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1445/2882] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 10:02:11 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 10:02:11.681755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d9fce3-12f2-43f0-bab0-7157a1f6e6f8 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:11.692682 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d9fce3-12f2-43f0-bab0-7157a1f6e6f8 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:11.697602 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c5e911-9c39-44a9-9d74-578b31a2fc7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:11.698284 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c5e911-9c39-44a9-9d74-578b31a2fc7a admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:11.698644 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c5e911-9c39-44a9-9d74-578b31a2fc7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:11.698927 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c5e911-9c39-44a9-9d74-578b31a2fc7a admin admin] PATH_INFO: `/v3/projects/7eecfa8e79144860a3def88b9bccd3d2/users/25aac992682b4805921580304671b67d/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:11.715356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c5e911-9c39-44a9-9d74-578b31a2fc7a admin admin] RBAC: Authorizing `identity:create_grant(project_id=7eecfa8e79144860a3def88b9bccd3d2, user_id=25aac992682b4805921580304671b67d, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:11.718383 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c5e911-9c39-44a9-9d74-578b31a2fc7a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:11.798457 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1438/2883] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 10:02:11 2022] PUT /identity/v3/projects/7eecfa8e79144860a3def88b9bccd3d2/users/25aac992682b4805921580304671b67d/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:11.808525 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bff341-8c90-42ee-a670-e42b60980e40 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:11.808828 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bff341-8c90-42ee-a670-e42b60980e40 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:11.809860 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bff341-8c90-42ee-a670-e42b60980e40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:11.836915 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-46bff341-8c90-42ee-a670-e42b60980e40 None None] MFA Rules not processed for user `25aac992682b4805921580304671b67d`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:11.838103 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-46bff341-8c90-42ee-a670-e42b60980e40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:11.901861 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1446/2884] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:11 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:02:11.906332 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c5e911-9c39-44a9-9d74-578b31a2fc7a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:11.925489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:11.971146 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c5e911-9c39-44a9-9d74-578b31a2fc7a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:11.977152 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cdcb1c-4551-48bc-a261-5063a00a0cc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:11.977755 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cdcb1c-4551-48bc-a261-5063a00a0cc6 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:11.978041 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cdcb1c-4551-48bc-a261-5063a00a0cc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:11.978310 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cdcb1c-4551-48bc-a261-5063a00a0cc6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:11.978981 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cdcb1c-4551-48bc-a261-5063a00a0cc6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:11.981669 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cdcb1c-4551-48bc-a261-5063a00a0cc6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:12.055404 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1439/2885] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 10:02:11 2022] POST /identity/v3/projects => generated 375 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 10:02:12.060084 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bff341-8c90-42ee-a670-e42b60980e40 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:12.070670 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bff341-8c90-42ee-a670-e42b60980e40 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:12.089949 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:12.125687 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7bd0e5-9fa9-449c-8c6e-a6cc153fc479 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:12.126283 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7bd0e5-9fa9-449c-8c6e-a6cc153fc479 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:12.126564 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7bd0e5-9fa9-449c-8c6e-a6cc153fc479 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:12.126859 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7bd0e5-9fa9-449c-8c6e-a6cc153fc479 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:12.127472 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7bd0e5-9fa9-449c-8c6e-a6cc153fc479 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:12.129398 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7bd0e5-9fa9-449c-8c6e-a6cc153fc479 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:12.210526 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1447/2886] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 10:02:12 2022] POST /identity/v3/users => generated 288 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 10:02:12.214491 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cdcb1c-4551-48bc-a261-5063a00a0cc6 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:12.225336 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cdcb1c-4551-48bc-a261-5063a00a0cc6 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:12.230042 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3371f9b5-48ee-4ea9-8f83-086e39b462cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:12.230669 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3371f9b5-48ee-4ea9-8f83-086e39b462cb admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:12.230963 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3371f9b5-48ee-4ea9-8f83-086e39b462cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:12.231912 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3371f9b5-48ee-4ea9-8f83-086e39b462cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:12.232553 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3371f9b5-48ee-4ea9-8f83-086e39b462cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:12.234152 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3371f9b5-48ee-4ea9-8f83-086e39b462cb admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:12.244641 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1440/2887] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 10:02:12 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 10:02:12.248530 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7bd0e5-9fa9-449c-8c6e-a6cc153fc479 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:12.258845 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7bd0e5-9fa9-449c-8c6e-a6cc153fc479 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:12.263509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4848eae8-c10d-4591-bfdc-66bad1547a99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:12.264097 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4848eae8-c10d-4591-bfdc-66bad1547a99 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:12.264459 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4848eae8-c10d-4591-bfdc-66bad1547a99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:12.264732 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4848eae8-c10d-4591-bfdc-66bad1547a99 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:12.265359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4848eae8-c10d-4591-bfdc-66bad1547a99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:12.266982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4848eae8-c10d-4591-bfdc-66bad1547a99 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:12.276657 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1448/2888] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 10:02:12 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 10:02:12.281106 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3371f9b5-48ee-4ea9-8f83-086e39b462cb admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:12.291073 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3371f9b5-48ee-4ea9-8f83-086e39b462cb admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:12.296319 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5536920c-623a-42c4-8b96-50b2fe115c4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:12.296963 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5536920c-623a-42c4-8b96-50b2fe115c4f admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:12.297248 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5536920c-623a-42c4-8b96-50b2fe115c4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:12.297533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5536920c-623a-42c4-8b96-50b2fe115c4f admin admin] PATH_INFO: `/v3/projects/562a7cabc2574abe8b59092a040ba0fd/users/095f492e51a143358f2315801c85084f/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:12.311875 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5536920c-623a-42c4-8b96-50b2fe115c4f admin admin] RBAC: Authorizing `identity:create_grant(project_id=562a7cabc2574abe8b59092a040ba0fd, user_id=095f492e51a143358f2315801c85084f, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:12.314489 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5536920c-623a-42c4-8b96-50b2fe115c4f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:12.380742 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1441/2889] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 10:02:12 2022] PUT /identity/v3/projects/562a7cabc2574abe8b59092a040ba0fd/users/095f492e51a143358f2315801c85084f/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 10:02:12.391932 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f732912e-867b-4a54-9db0-94e5f1a30b03 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:12.392222 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f732912e-867b-4a54-9db0-94e5f1a30b03 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:12.392496 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f732912e-867b-4a54-9db0-94e5f1a30b03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:12.421523 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-f732912e-867b-4a54-9db0-94e5f1a30b03 None None] MFA Rules not processed for user `095f492e51a143358f2315801c85084f`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:12.422751 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-f732912e-867b-4a54-9db0-94e5f1a30b03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:12.484420 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1449/2890] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:12 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:02:12.488863 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5536920c-623a-42c4-8b96-50b2fe115c4f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:12.549512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5536920c-623a-42c4-8b96-50b2fe115c4f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:12.554150 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bedd9b6-3e82-47f7-b2c8-6f661ef4a343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:12.554784 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bedd9b6-3e82-47f7-b2c8-6f661ef4a343 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:12.555062 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bedd9b6-3e82-47f7-b2c8-6f661ef4a343 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:12.555330 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bedd9b6-3e82-47f7-b2c8-6f661ef4a343 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:12.555951 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bedd9b6-3e82-47f7-b2c8-6f661ef4a343 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:12.557860 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bedd9b6-3e82-47f7-b2c8-6f661ef4a343 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:12.679306 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1442/2891] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 10:02:12 2022] POST /identity/v3/projects => generated 375 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 10:02:12.683243 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f732912e-867b-4a54-9db0-94e5f1a30b03 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:12.693617 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f732912e-867b-4a54-9db0-94e5f1a30b03 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:12.749654 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0326bf55-95a3-4a1f-992a-847ea79e5844 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:12.750255 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0326bf55-95a3-4a1f-992a-847ea79e5844 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:12.750537 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0326bf55-95a3-4a1f-992a-847ea79e5844 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:12.750839 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0326bf55-95a3-4a1f-992a-847ea79e5844 admin admin] PATH_INFO: `/v3/users` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:12.751470 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0326bf55-95a3-4a1f-992a-847ea79e5844 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:12.753435 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0326bf55-95a3-4a1f-992a-847ea79e5844 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:12.827656 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1450/2892] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 10:02:12 2022] POST /identity/v3/users => generated 294 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 10:02:12.831623 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bedd9b6-3e82-47f7-b2c8-6f661ef4a343 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:12.841807 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bedd9b6-3e82-47f7-b2c8-6f661ef4a343 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:12.847129 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0042d42f-9b73-443b-ac67-b2b90816e131 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:12.847714 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0042d42f-9b73-443b-ac67-b2b90816e131 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:12.847990 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0042d42f-9b73-443b-ac67-b2b90816e131 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:12.848260 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0042d42f-9b73-443b-ac67-b2b90816e131 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:12.848912 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0042d42f-9b73-443b-ac67-b2b90816e131 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:12.850495 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0042d42f-9b73-443b-ac67-b2b90816e131 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:12.860885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1443/2893] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 10:02:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 10:02:12.864851 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0326bf55-95a3-4a1f-992a-847ea79e5844 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:12.874712 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0326bf55-95a3-4a1f-992a-847ea79e5844 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:12.879230 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43495d5-b0ba-49be-a426-58dc9e94cd41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:12.879868 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43495d5-b0ba-49be-a426-58dc9e94cd41 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:12.880146 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43495d5-b0ba-49be-a426-58dc9e94cd41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:12.880418 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43495d5-b0ba-49be-a426-58dc9e94cd41 admin admin] PATH_INFO: `/v3/projects/4c363685018e4809abd8c16a4e1372da/users/e10294a50c99465b8f3e1af9cda72b7c/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:12.895533 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43495d5-b0ba-49be-a426-58dc9e94cd41 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c363685018e4809abd8c16a4e1372da, user_id=e10294a50c99465b8f3e1af9cda72b7c, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:12.898193 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43495d5-b0ba-49be-a426-58dc9e94cd41 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:12.954134 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:12.959648 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1451/2894] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 10:02:12 2022] PUT /identity/v3/projects/4c363685018e4809abd8c16a4e1372da/users/e10294a50c99465b8f3e1af9cda72b7c/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:12.963534 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0042d42f-9b73-443b-ac67-b2b90816e131 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:12.973386 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0042d42f-9b73-443b-ac67-b2b90816e131 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:13.011052 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:13.029226 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d2bf48-8018-4e82-8596-74082fae964e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:13.029848 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d2bf48-8018-4e82-8596-74082fae964e admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:13.030125 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d2bf48-8018-4e82-8596-74082fae964e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:13.030524 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d2bf48-8018-4e82-8596-74082fae964e admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:13.031182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d2bf48-8018-4e82-8596-74082fae964e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:13.032851 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d2bf48-8018-4e82-8596-74082fae964e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:13.042930 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1444/2895] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 10:02:12 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 10:02:13.047281 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43495d5-b0ba-49be-a426-58dc9e94cd41 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:13.057295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43495d5-b0ba-49be-a426-58dc9e94cd41 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:13.062955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06c9383-ec5d-431c-a233-f0a8de686e90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:13.063690 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06c9383-ec5d-431c-a233-f0a8de686e90 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:13.064009 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06c9383-ec5d-431c-a233-f0a8de686e90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:13.064465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06c9383-ec5d-431c-a233-f0a8de686e90 admin admin] PATH_INFO: `/v3/projects/4c363685018e4809abd8c16a4e1372da/users/e10294a50c99465b8f3e1af9cda72b7c/roles/e874c311084c4c9fb7691bce461d319b` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:13.066333 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06c9383-ec5d-431c-a233-f0a8de686e90 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c363685018e4809abd8c16a4e1372da, user_id=e10294a50c99465b8f3e1af9cda72b7c, role_id=e874c311084c4c9fb7691bce461d319b)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:13.069000 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06c9383-ec5d-431c-a233-f0a8de686e90 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:13.079801 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1452/2896] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 10:02:13 2022] PUT /identity/v3/projects/4c363685018e4809abd8c16a4e1372da/users/e10294a50c99465b8f3e1af9cda72b7c/roles/e874c311084c4c9fb7691bce461d319b => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:13.089575 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f68b204-b190-4fc3-8549-b6b531fd6b6e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:13.089862 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f68b204-b190-4fc3-8549-b6b531fd6b6e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:13.090136 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f68b204-b190-4fc3-8549-b6b531fd6b6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:13.117329 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-2f68b204-b190-4fc3-8549-b6b531fd6b6e None None] MFA Rules not processed for user `e10294a50c99465b8f3e1af9cda72b7c`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:13.123003 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2f68b204-b190-4fc3-8549-b6b531fd6b6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:13.187355 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1445/2897] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:13 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:02:13.198685 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06c9383-ec5d-431c-a233-f0a8de686e90 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:13.200478 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-b06c9383-ec5d-431c-a233-f0a8de686e90 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:13.257296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06c9383-ec5d-431c-a233-f0a8de686e90 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:13.260970 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebc8922-96e8-4a04-89fa-0c6bb324ad8a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:13.261564 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebc8922-96e8-4a04-89fa-0c6bb324ad8a service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:13.261840 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebc8922-96e8-4a04-89fa-0c6bb324ad8a service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:13.262106 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebc8922-96e8-4a04-89fa-0c6bb324ad8a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:13.317820 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebc8922-96e8-4a04-89fa-0c6bb324ad8a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:13.319295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebc8922-96e8-4a04-89fa-0c6bb324ad8a service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:13.328039 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1453/2898] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 10:02:13 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 10:02:15.189709 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f68b204-b190-4fc3-8549-b6b531fd6b6e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:15.194216 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2f68b204-b190-4fc3-8549-b6b531fd6b6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:15.203533 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f68b204-b190-4fc3-8549-b6b531fd6b6e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:15.207020 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493ba57a-3771-44a1-bbc2-619b12888aed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:15.207621 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493ba57a-3771-44a1-bbc2-619b12888aed service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:15.207896 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493ba57a-3771-44a1-bbc2-619b12888aed service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:15.208164 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493ba57a-3771-44a1-bbc2-619b12888aed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:15.265195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493ba57a-3771-44a1-bbc2-619b12888aed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:15.266676 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493ba57a-3771-44a1-bbc2-619b12888aed service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:15.275829 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1446/2899] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 10:02:15 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 10:02:16.546691 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebc8922-96e8-4a04-89fa-0c6bb324ad8a service cinder] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:16.555351 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebc8922-96e8-4a04-89fa-0c6bb324ad8a service cinder] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:16.559043 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5adc965-0e79-4866-8846-87bd9522dc52 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:16.559633 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5adc965-0e79-4866-8846-87bd9522dc52 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:16.559914 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5adc965-0e79-4866-8846-87bd9522dc52 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:16.560186 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5adc965-0e79-4866-8846-87bd9522dc52 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:16.569614 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5adc965-0e79-4866-8846-87bd9522dc52 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:16.571094 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5adc965-0e79-4866-8846-87bd9522dc52 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:16.580298 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1454/2900] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 10:02:16 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 10:02:21.036203 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493ba57a-3771-44a1-bbc2-619b12888aed service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:21.098415 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493ba57a-3771-44a1-bbc2-619b12888aed service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:21.103959 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684d264d-988d-4abb-b502-38722e6cc186 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:21.104688 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684d264d-988d-4abb-b502-38722e6cc186 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:21.104962 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684d264d-988d-4abb-b502-38722e6cc186 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:21.105498 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684d264d-988d-4abb-b502-38722e6cc186 admin admin] PATH_INFO: `/v3/users/25aac992682b4805921580304671b67d` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:21.106849 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684d264d-988d-4abb-b502-38722e6cc186 admin admin] RBAC: Authorizing `identity:delete_user(user_id=25aac992682b4805921580304671b67d)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:21.108836 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684d264d-988d-4abb-b502-38722e6cc186 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:21.112302 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:21.330993 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-684d264d-988d-4abb-b502-38722e6cc186 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '25aac992682b4805921580304671b67d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:21.331652 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-684d264d-988d-4abb-b502-38722e6cc186 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '25aac992682b4805921580304671b67d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:21.332247 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-684d264d-988d-4abb-b502-38722e6cc186 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '25aac992682b4805921580304671b67d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:21.339083 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-684d264d-988d-4abb-b502-38722e6cc186 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '25aac992682b4805921580304671b67d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:21.415180 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-684d264d-988d-4abb-b502-38722e6cc186 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '25aac992682b4805921580304671b67d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:21.432802 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1447/2901] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 10:02:21 2022] DELETE /identity/v3/users/25aac992682b4805921580304671b67d => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:21.437372 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926658e8-bc78-42d7-b4d6-b09636b639c4 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:21.437838 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926658e8-bc78-42d7-b4d6-b09636b639c4 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:21.439148 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926658e8-bc78-42d7-b4d6-b09636b639c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:21.457521 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-926658e8-bc78-42d7-b4d6-b09636b639c4 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:21.458703 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-926658e8-bc78-42d7-b4d6-b09636b639c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:21.519884 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1455/2902] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:02:21.529624 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684d264d-988d-4abb-b502-38722e6cc186 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:21.531074 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-684d264d-988d-4abb-b502-38722e6cc186 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:21.590422 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684d264d-988d-4abb-b502-38722e6cc186 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:21.594544 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b599c68a-75d3-477a-96c0-59c4216fd4a2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:21.595200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b599c68a-75d3-477a-96c0-59c4216fd4a2 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:21.595478 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b599c68a-75d3-477a-96c0-59c4216fd4a2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:21.595747 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b599c68a-75d3-477a-96c0-59c4216fd4a2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:21.605161 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b599c68a-75d3-477a-96c0-59c4216fd4a2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:21.606675 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b599c68a-75d3-477a-96c0-59c4216fd4a2 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:21.616195 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1448/2903] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 10:02:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 10:02:22.010972 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926658e8-bc78-42d7-b4d6-b09636b639c4 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:22.013017 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-926658e8-bc78-42d7-b4d6-b09636b639c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:22.024812 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926658e8-bc78-42d7-b4d6-b09636b639c4 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:22.029415 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1de0af3-4d46-4571-a03b-bb62060581d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:22.030038 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1de0af3-4d46-4571-a03b-bb62060581d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:22.030317 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1de0af3-4d46-4571-a03b-bb62060581d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:22.030611 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1de0af3-4d46-4571-a03b-bb62060581d9 admin admin] PATH_INFO: `/v3/projects/7eecfa8e79144860a3def88b9bccd3d2` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:22.031787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1de0af3-4d46-4571-a03b-bb62060581d9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7eecfa8e79144860a3def88b9bccd3d2)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:22.033704 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1de0af3-4d46-4571-a03b-bb62060581d9 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:22.056489 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:22.129412 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-d1de0af3-4d46-4571-a03b-bb62060581d9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7eecfa8e79144860a3def88b9bccd3d2', 'request_id': 'req-d1de0af3-4d46-4571-a03b-bb62060581d9'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:22.135431 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-d1de0af3-4d46-4571-a03b-bb62060581d9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7eecfa8e79144860a3def88b9bccd3d2', 'request_id': 'req-d1de0af3-4d46-4571-a03b-bb62060581d9'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:22.142122 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-d1de0af3-4d46-4571-a03b-bb62060581d9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7eecfa8e79144860a3def88b9bccd3d2', 'request_id': 'req-d1de0af3-4d46-4571-a03b-bb62060581d9'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:22.214521 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-d1de0af3-4d46-4571-a03b-bb62060581d9 admin admin] The token cache is being invalidate because project 7eecfa8e79144860a3def88b9bccd3d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 10:02:22.217775 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-d1de0af3-4d46-4571-a03b-bb62060581d9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:22.217775 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-d1de0af3-4d46-4571-a03b-bb62060581d9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:22.217775 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1456/2904] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 10:02:22 2022] DELETE /identity/v3/projects/7eecfa8e79144860a3def88b9bccd3d2 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:22.220999 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b599c68a-75d3-477a-96c0-59c4216fd4a2 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:22.230553 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b599c68a-75d3-477a-96c0-59c4216fd4a2 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:22.285763 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3f423c-1957-4a75-8486-c98f1f3a6fec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:22.286313 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3f423c-1957-4a75-8486-c98f1f3a6fec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:22.286613 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3f423c-1957-4a75-8486-c98f1f3a6fec admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:22.286890 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3f423c-1957-4a75-8486-c98f1f3a6fec admin admin] PATH_INFO: `/v3/users/095f492e51a143358f2315801c85084f` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:22.287985 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3f423c-1957-4a75-8486-c98f1f3a6fec admin admin] RBAC: Authorizing `identity:delete_user(user_id=095f492e51a143358f2315801c85084f)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:22.289885 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3f423c-1957-4a75-8486-c98f1f3a6fec admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:22.311807 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:22.461178 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0d3f423c-1957-4a75-8486-c98f1f3a6fec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '095f492e51a143358f2315801c85084f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:22.461813 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0d3f423c-1957-4a75-8486-c98f1f3a6fec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '095f492e51a143358f2315801c85084f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:22.462403 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0d3f423c-1957-4a75-8486-c98f1f3a6fec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '095f492e51a143358f2315801c85084f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:22.469182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0d3f423c-1957-4a75-8486-c98f1f3a6fec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '095f492e51a143358f2315801c85084f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:22.557730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0d3f423c-1957-4a75-8486-c98f1f3a6fec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '095f492e51a143358f2315801c85084f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:22.576069 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1449/2905] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 10:02:22 2022] DELETE /identity/v3/users/095f492e51a143358f2315801c85084f => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:22.935892 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1de0af3-4d46-4571-a03b-bb62060581d9 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:22.940809 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d1de0af3-4d46-4571-a03b-bb62060581d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:22.952409 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1de0af3-4d46-4571-a03b-bb62060581d9 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:23.009306 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef0eeb4-ee4d-44ac-9536-6c832b2c153c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:23.009930 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef0eeb4-ee4d-44ac-9536-6c832b2c153c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:23.010207 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef0eeb4-ee4d-44ac-9536-6c832b2c153c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:23.010474 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef0eeb4-ee4d-44ac-9536-6c832b2c153c admin admin] PATH_INFO: `/v3/projects/562a7cabc2574abe8b59092a040ba0fd` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:23.011591 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef0eeb4-ee4d-44ac-9536-6c832b2c153c admin admin] RBAC: Authorizing `identity:delete_project(project_id=562a7cabc2574abe8b59092a040ba0fd)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:23.013492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef0eeb4-ee4d-44ac-9536-6c832b2c153c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:23.112655 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-1ef0eeb4-ee4d-44ac-9536-6c832b2c153c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '562a7cabc2574abe8b59092a040ba0fd', 'request_id': 'req-1ef0eeb4-ee4d-44ac-9536-6c832b2c153c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:23.118350 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-1ef0eeb4-ee4d-44ac-9536-6c832b2c153c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '562a7cabc2574abe8b59092a040ba0fd', 'request_id': 'req-1ef0eeb4-ee4d-44ac-9536-6c832b2c153c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:23.124450 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-1ef0eeb4-ee4d-44ac-9536-6c832b2c153c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '562a7cabc2574abe8b59092a040ba0fd', 'request_id': 'req-1ef0eeb4-ee4d-44ac-9536-6c832b2c153c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:23.203570 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-1ef0eeb4-ee4d-44ac-9536-6c832b2c153c admin admin] The token cache is being invalidate because project 562a7cabc2574abe8b59092a040ba0fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 10:02:23.203865 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-1ef0eeb4-ee4d-44ac-9536-6c832b2c153c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:23.204492 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-1ef0eeb4-ee4d-44ac-9536-6c832b2c153c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:23.206007 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1457/2906] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 10:02:22 2022] DELETE /identity/v3/projects/562a7cabc2574abe8b59092a040ba0fd => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:23.210118 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3f423c-1957-4a75-8486-c98f1f3a6fec admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:23.211580 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0d3f423c-1957-4a75-8486-c98f1f3a6fec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:23.224810 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3f423c-1957-4a75-8486-c98f1f3a6fec admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:23.279300 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7cf7d9-ded2-40f4-a8b8-f8cd868c32c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:23.279963 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7cf7d9-ded2-40f4-a8b8-f8cd868c32c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:23.280246 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7cf7d9-ded2-40f4-a8b8-f8cd868c32c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:23.280522 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7cf7d9-ded2-40f4-a8b8-f8cd868c32c5 admin admin] PATH_INFO: `/v3/users/e10294a50c99465b8f3e1af9cda72b7c` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:23.282281 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7cf7d9-ded2-40f4-a8b8-f8cd868c32c5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e10294a50c99465b8f3e1af9cda72b7c)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:23.284266 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7cf7d9-ded2-40f4-a8b8-f8cd868c32c5 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:23.451218 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ff7cf7d9-ded2-40f4-a8b8-f8cd868c32c5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e10294a50c99465b8f3e1af9cda72b7c'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:23.451869 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ff7cf7d9-ded2-40f4-a8b8-f8cd868c32c5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e10294a50c99465b8f3e1af9cda72b7c'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:23.452474 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ff7cf7d9-ded2-40f4-a8b8-f8cd868c32c5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e10294a50c99465b8f3e1af9cda72b7c'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:23.460284 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ff7cf7d9-ded2-40f4-a8b8-f8cd868c32c5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e10294a50c99465b8f3e1af9cda72b7c'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:23.500737 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ff7cf7d9-ded2-40f4-a8b8-f8cd868c32c5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e10294a50c99465b8f3e1af9cda72b7c'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:23.518739 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1450/2907] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 10:02:23 2022] DELETE /identity/v3/users/e10294a50c99465b8f3e1af9cda72b7c => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:23.908279 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef0eeb4-ee4d-44ac-9536-6c832b2c153c admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:23.912398 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-1ef0eeb4-ee4d-44ac-9536-6c832b2c153c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:23.923041 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef0eeb4-ee4d-44ac-9536-6c832b2c153c admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:23.961278 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:23.978835 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7b33e7-fef6-4f01-ae22-cd98a8856ec0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:23.979507 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7b33e7-fef6-4f01-ae22-cd98a8856ec0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:23.979787 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7b33e7-fef6-4f01-ae22-cd98a8856ec0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:23.980061 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7b33e7-fef6-4f01-ae22-cd98a8856ec0 admin admin] PATH_INFO: `/v3/projects/4c363685018e4809abd8c16a4e1372da` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:23.981163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7b33e7-fef6-4f01-ae22-cd98a8856ec0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4c363685018e4809abd8c16a4e1372da)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:23.983115 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7b33e7-fef6-4f01-ae22-cd98a8856ec0 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:24.088972 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8a7b33e7-fef6-4f01-ae22-cd98a8856ec0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4c363685018e4809abd8c16a4e1372da', 'request_id': 'req-8a7b33e7-fef6-4f01-ae22-cd98a8856ec0'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:24.094864 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8a7b33e7-fef6-4f01-ae22-cd98a8856ec0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4c363685018e4809abd8c16a4e1372da', 'request_id': 'req-8a7b33e7-fef6-4f01-ae22-cd98a8856ec0'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:24.100883 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8a7b33e7-fef6-4f01-ae22-cd98a8856ec0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4c363685018e4809abd8c16a4e1372da', 'request_id': 'req-8a7b33e7-fef6-4f01-ae22-cd98a8856ec0'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:24.180293 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8a7b33e7-fef6-4f01-ae22-cd98a8856ec0 admin admin] The token cache is being invalidate because project 4c363685018e4809abd8c16a4e1372da was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 10:02:24.180718 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8a7b33e7-fef6-4f01-ae22-cd98a8856ec0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:24.181408 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-8a7b33e7-fef6-4f01-ae22-cd98a8856ec0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:24.182941 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1458/2908] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 10:02:23 2022] DELETE /identity/v3/projects/4c363685018e4809abd8c16a4e1372da => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:24.187842 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8deeb91-4bed-46a2-b43d-4045d5c997d5 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:24.193034 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8deeb91-4bed-46a2-b43d-4045d5c997d5 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:24.193034 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8deeb91-4bed-46a2-b43d-4045d5c997d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:24.208518 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e8deeb91-4bed-46a2-b43d-4045d5c997d5 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:24.209730 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e8deeb91-4bed-46a2-b43d-4045d5c997d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:24.223707 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:24.269444 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1451/2909] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:02:24.277244 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab91f0c6-d8b7-49dc-afbd-ef147478ebf6 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:24.280654 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab91f0c6-d8b7-49dc-afbd-ef147478ebf6 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:24.280654 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab91f0c6-d8b7-49dc-afbd-ef147478ebf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:24.298856 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-ab91f0c6-d8b7-49dc-afbd-ef147478ebf6 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:24.300060 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-ab91f0c6-d8b7-49dc-afbd-ef147478ebf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:24.308268 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1459/2910] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:02:24.312591 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8deeb91-4bed-46a2-b43d-4045d5c997d5 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:24.322371 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8deeb91-4bed-46a2-b43d-4045d5c997d5 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:24.327959 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c46fbd-5b21-42f1-8a6c-09d9a52f6f42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:24.328562 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c46fbd-5b21-42f1-8a6c-09d9a52f6f42 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:24.328839 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c46fbd-5b21-42f1-8a6c-09d9a52f6f42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:24.329109 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c46fbd-5b21-42f1-8a6c-09d9a52f6f42 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:24.329742 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c46fbd-5b21-42f1-8a6c-09d9a52f6f42 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 10:02:24.330040 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c46fbd-5b21-42f1-8a6c-09d9a52f6f42 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:24.331660 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c46fbd-5b21-42f1-8a6c-09d9a52f6f42 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:24.343119 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1452/2911] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 10:02:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 10:02:24.347953 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab91f0c6-d8b7-49dc-afbd-ef147478ebf6 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:24.357286 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab91f0c6-d8b7-49dc-afbd-ef147478ebf6 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:24.362468 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d2854e-0e9e-439f-9a73-524c1d8e885b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:24.363095 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d2854e-0e9e-439f-9a73-524c1d8e885b admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:24.363376 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d2854e-0e9e-439f-9a73-524c1d8e885b admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:24.363649 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d2854e-0e9e-439f-9a73-524c1d8e885b admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:24.364276 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d2854e-0e9e-439f-9a73-524c1d8e885b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:24.366211 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d2854e-0e9e-439f-9a73-524c1d8e885b admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:24.473359 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1460/2912] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 10:02:24 2022] POST /identity/v3/projects => generated 371 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 10:02:24.477683 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c46fbd-5b21-42f1-8a6c-09d9a52f6f42 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:24.488081 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c46fbd-5b21-42f1-8a6c-09d9a52f6f42 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:24.545592 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb50884-ea20-4cef-9bbb-c3de4d89e20f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:24.546196 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb50884-ea20-4cef-9bbb-c3de4d89e20f admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:24.546505 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb50884-ea20-4cef-9bbb-c3de4d89e20f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:24.547315 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb50884-ea20-4cef-9bbb-c3de4d89e20f admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:24.547947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb50884-ea20-4cef-9bbb-c3de4d89e20f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:24.549842 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb50884-ea20-4cef-9bbb-c3de4d89e20f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:24.632593 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1453/2913] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 10:02:24 2022] POST /identity/v3/users => generated 286 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 10:02:24.636827 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d2854e-0e9e-439f-9a73-524c1d8e885b admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:24.647296 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d2854e-0e9e-439f-9a73-524c1d8e885b admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:24.651864 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1cd1ca-b0ec-4a3b-9439-1cda5b4cfed6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:24.652440 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1cd1ca-b0ec-4a3b-9439-1cda5b4cfed6 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:24.652715 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1cd1ca-b0ec-4a3b-9439-1cda5b4cfed6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:24.652983 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1cd1ca-b0ec-4a3b-9439-1cda5b4cfed6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:24.653688 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1cd1ca-b0ec-4a3b-9439-1cda5b4cfed6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:24.655315 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1cd1ca-b0ec-4a3b-9439-1cda5b4cfed6 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:24.665295 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1461/2914] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 10:02:24 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 10:02:24.669148 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb50884-ea20-4cef-9bbb-c3de4d89e20f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:24.679586 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb50884-ea20-4cef-9bbb-c3de4d89e20f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:24.684291 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1941e71e-0c09-440b-a43e-eba5f0d2a8ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:24.684903 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1941e71e-0c09-440b-a43e-eba5f0d2a8ea admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:24.685182 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1941e71e-0c09-440b-a43e-eba5f0d2a8ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:24.685449 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1941e71e-0c09-440b-a43e-eba5f0d2a8ea admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:24.686075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1941e71e-0c09-440b-a43e-eba5f0d2a8ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:24.688356 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1941e71e-0c09-440b-a43e-eba5f0d2a8ea admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:24.698064 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1454/2915] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 10:02:24 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 10:02:24.702064 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1cd1ca-b0ec-4a3b-9439-1cda5b4cfed6 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:24.711929 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1cd1ca-b0ec-4a3b-9439-1cda5b4cfed6 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:24.716499 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003ab152-a8e7-4747-a527-989de6f02594 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:24.717123 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003ab152-a8e7-4747-a527-989de6f02594 admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:24.717401 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003ab152-a8e7-4747-a527-989de6f02594 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:24.717671 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003ab152-a8e7-4747-a527-989de6f02594 admin admin] PATH_INFO: `/v3/projects/52cdb6161cd54d0b9330532d7c6d4029/users/4b7a56303a5848eb964eaf81e3546c14/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:24.735114 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003ab152-a8e7-4747-a527-989de6f02594 admin admin] RBAC: Authorizing `identity:create_grant(project_id=52cdb6161cd54d0b9330532d7c6d4029, user_id=4b7a56303a5848eb964eaf81e3546c14, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:24.737701 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003ab152-a8e7-4747-a527-989de6f02594 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:24.792113 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1462/2916] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 10:02:24 2022] PUT /identity/v3/projects/52cdb6161cd54d0b9330532d7c6d4029/users/4b7a56303a5848eb964eaf81e3546c14/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:24.803837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f60708-62af-46b8-98f6-1c9ae172053d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:24.806159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f60708-62af-46b8-98f6-1c9ae172053d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:24.806159 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f60708-62af-46b8-98f6-1c9ae172053d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:24.832171 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-38f60708-62af-46b8-98f6-1c9ae172053d None None] MFA Rules not processed for user `4b7a56303a5848eb964eaf81e3546c14`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:24.833389 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-38f60708-62af-46b8-98f6-1c9ae172053d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:24.895628 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1455/2917] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:24 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:02:24.907911 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003ab152-a8e7-4747-a527-989de6f02594 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:24.909605 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-003ab152-a8e7-4747-a527-989de6f02594 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:24.981757 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:24.987465 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003ab152-a8e7-4747-a527-989de6f02594 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:24.991208 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f36166-18bd-49fd-95ad-aa13ad567d3d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf907d4033064fb8a2707b1bbb0a180b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:24.994968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f36166-18bd-49fd-95ad-aa13ad567d3d service glance] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:24.994968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f36166-18bd-49fd-95ad-aa13ad567d3d service glance] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:24.994968 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f36166-18bd-49fd-95ad-aa13ad567d3d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:25.001051 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f36166-18bd-49fd-95ad-aa13ad567d3d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:25.002536 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f36166-18bd-49fd-95ad-aa13ad567d3d service glance] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:25.012379 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1463/2918] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 10:02:24 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 10:02:34.892679 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=90,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:35.123284 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=5 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:53.092048 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f60708-62af-46b8-98f6-1c9ae172053d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:53.152858 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f60708-62af-46b8-98f6-1c9ae172053d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:53.157393 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e25fbb-eb13-4ba9-998b-6b3b69b4e663 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:53.158016 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e25fbb-eb13-4ba9-998b-6b3b69b4e663 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:53.158297 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e25fbb-eb13-4ba9-998b-6b3b69b4e663 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:53.158572 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e25fbb-eb13-4ba9-998b-6b3b69b4e663 admin admin] PATH_INFO: `/v3/users/4b7a56303a5848eb964eaf81e3546c14` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:53.159696 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e25fbb-eb13-4ba9-998b-6b3b69b4e663 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4b7a56303a5848eb964eaf81e3546c14)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:53.162134 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e25fbb-eb13-4ba9-998b-6b3b69b4e663 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:53.330413 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d0e25fbb-eb13-4ba9-998b-6b3b69b4e663 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4b7a56303a5848eb964eaf81e3546c14'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:53.331014 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d0e25fbb-eb13-4ba9-998b-6b3b69b4e663 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4b7a56303a5848eb964eaf81e3546c14'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:53.331650 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d0e25fbb-eb13-4ba9-998b-6b3b69b4e663 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4b7a56303a5848eb964eaf81e3546c14'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:53.338512 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d0e25fbb-eb13-4ba9-998b-6b3b69b4e663 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4b7a56303a5848eb964eaf81e3546c14'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:53.391245 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d0e25fbb-eb13-4ba9-998b-6b3b69b4e663 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4b7a56303a5848eb964eaf81e3546c14'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:53.408287 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1456/2919] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 10:02:53 2022] DELETE /identity/v3/users/4b7a56303a5848eb964eaf81e3546c14 => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:53.413391 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d028648e-8599-4def-aa0b-16e8781ed4a7 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:53.413873 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d028648e-8599-4def-aa0b-16e8781ed4a7 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:53.413988 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d028648e-8599-4def-aa0b-16e8781ed4a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:53.433138 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-d028648e-8599-4def-aa0b-16e8781ed4a7 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:53.434339 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d028648e-8599-4def-aa0b-16e8781ed4a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:53.492032 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1464/2920] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:02:53.501546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e25fbb-eb13-4ba9-998b-6b3b69b4e663 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:53.502996 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d0e25fbb-eb13-4ba9-998b-6b3b69b4e663 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:53.559923 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e25fbb-eb13-4ba9-998b-6b3b69b4e663 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:53.563499 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bbd8fe-f627-4d61-83e2-5aed0d7c532e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:53.564093 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bbd8fe-f627-4d61-83e2-5aed0d7c532e service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:53.564367 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bbd8fe-f627-4d61-83e2-5aed0d7c532e service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:53.564635 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bbd8fe-f627-4d61-83e2-5aed0d7c532e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:53.573368 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bbd8fe-f627-4d61-83e2-5aed0d7c532e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:53.574807 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bbd8fe-f627-4d61-83e2-5aed0d7c532e service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:53.584837 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1457/2921] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 10:02:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 10:02:53.975540 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d028648e-8599-4def-aa0b-16e8781ed4a7 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:53.977107 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-d028648e-8599-4def-aa0b-16e8781ed4a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:53.989476 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d028648e-8599-4def-aa0b-16e8781ed4a7 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:53.994504 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021612a1-30aa-46d0-a957-5602d622ab4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:53.995168 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021612a1-30aa-46d0-a957-5602d622ab4c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:53.995474 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021612a1-30aa-46d0-a957-5602d622ab4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:53.995728 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021612a1-30aa-46d0-a957-5602d622ab4c admin admin] PATH_INFO: `/v3/projects/52cdb6161cd54d0b9330532d7c6d4029` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:53.996837 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021612a1-30aa-46d0-a957-5602d622ab4c admin admin] RBAC: Authorizing `identity:delete_project(project_id=52cdb6161cd54d0b9330532d7c6d4029)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:53.998807 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021612a1-30aa-46d0-a957-5602d622ab4c admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:54.128427 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-021612a1-30aa-46d0-a957-5602d622ab4c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '52cdb6161cd54d0b9330532d7c6d4029', 'request_id': 'req-021612a1-30aa-46d0-a957-5602d622ab4c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:54.134834 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-021612a1-30aa-46d0-a957-5602d622ab4c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '52cdb6161cd54d0b9330532d7c6d4029', 'request_id': 'req-021612a1-30aa-46d0-a957-5602d622ab4c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:54.141280 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-021612a1-30aa-46d0-a957-5602d622ab4c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '52cdb6161cd54d0b9330532d7c6d4029', 'request_id': 'req-021612a1-30aa-46d0-a957-5602d622ab4c'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:54.202657 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-021612a1-30aa-46d0-a957-5602d622ab4c admin admin] The token cache is being invalidate because project 52cdb6161cd54d0b9330532d7c6d4029 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 10:02:54.202955 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-021612a1-30aa-46d0-a957-5602d622ab4c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:54.208028 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-021612a1-30aa-46d0-a957-5602d622ab4c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:02:54.208028 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1465/2922] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 10:02:53 2022] DELETE /identity/v3/projects/52cdb6161cd54d0b9330532d7c6d4029 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:02:54.210420 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edb5224-a0d9-447a-904b-52bcd29dbe31 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:54.210903 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edb5224-a0d9-447a-904b-52bcd29dbe31 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:54.211019 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edb5224-a0d9-447a-904b-52bcd29dbe31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:54.230044 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-6edb5224-a0d9-447a-904b-52bcd29dbe31 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:54.231260 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6edb5224-a0d9-447a-904b-52bcd29dbe31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:54.247977 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:54.290831 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1458/2923] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 30 10:02:54.298560 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b176a3-7381-4ffc-ab9a-e0c74d3eeab6 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:54.298886 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b176a3-7381-4ffc-ab9a-e0c74d3eeab6 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:54.299159 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b176a3-7381-4ffc-ab9a-e0c74d3eeab6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:54.320612 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-23b176a3-7381-4ffc-ab9a-e0c74d3eeab6 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:54.321827 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-23b176a3-7381-4ffc-ab9a-e0c74d3eeab6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:54.329827 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1466/2924] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:02:54.334075 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edb5224-a0d9-447a-904b-52bcd29dbe31 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:54.343868 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edb5224-a0d9-447a-904b-52bcd29dbe31 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:54.348470 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a275d4-825f-4cc7-a210-8763fd177d62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:54.349071 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a275d4-825f-4cc7-a210-8763fd177d62 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:54.349349 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a275d4-825f-4cc7-a210-8763fd177d62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:54.349625 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a275d4-825f-4cc7-a210-8763fd177d62 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:54.350562 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a275d4-825f-4cc7-a210-8763fd177d62 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 30 10:02:54.350947 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a275d4-825f-4cc7-a210-8763fd177d62 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:54.353168 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a275d4-825f-4cc7-a210-8763fd177d62 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:54.365023 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1459/2925] 10.222.0.39 () {58 vars in 1153 bytes} [Fri Sep 30 10:02:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 30 10:02:54.369303 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b176a3-7381-4ffc-ab9a-e0c74d3eeab6 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:54.378310 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b176a3-7381-4ffc-ab9a-e0c74d3eeab6 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:54.383975 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1ae675-8b09-477f-8205-91c02e5cdf17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:54.384502 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1ae675-8b09-477f-8205-91c02e5cdf17 admin admin] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:54.384778 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1ae675-8b09-477f-8205-91c02e5cdf17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:54.385048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1ae675-8b09-477f-8205-91c02e5cdf17 admin admin] PATH_INFO: `/v3/projects` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:54.385668 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1ae675-8b09-477f-8205-91c02e5cdf17 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:54.387600 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1ae675-8b09-477f-8205-91c02e5cdf17 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:54.472338 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1467/2926] 10.222.0.39 () {60 vars in 1153 bytes} [Fri Sep 30 10:02:54 2022] POST /identity/v3/projects => generated 375 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 10:02:54.476498 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a275d4-825f-4cc7-a210-8763fd177d62 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:54.486315 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a275d4-825f-4cc7-a210-8763fd177d62 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:54.541943 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5df758-a267-4d97-8f5a-a6e910403ad0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:54.542546 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5df758-a267-4d97-8f5a-a6e910403ad0 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:54.542860 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5df758-a267-4d97-8f5a-a6e910403ad0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:54.543141 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5df758-a267-4d97-8f5a-a6e910403ad0 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:54.543758 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5df758-a267-4d97-8f5a-a6e910403ad0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:54.545739 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5df758-a267-4d97-8f5a-a6e910403ad0 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:54.642508 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1460/2927] 10.222.0.39 () {60 vars in 1144 bytes} [Fri Sep 30 10:02:54 2022] POST /identity/v3/users => generated 288 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 30 10:02:54.646640 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1ae675-8b09-477f-8205-91c02e5cdf17 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:54.656755 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1ae675-8b09-477f-8205-91c02e5cdf17 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:54.662335 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a1546b-698d-4ed0-8b38-69bdfc2300e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:54.662977 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a1546b-698d-4ed0-8b38-69bdfc2300e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:54.663260 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a1546b-698d-4ed0-8b38-69bdfc2300e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:54.663566 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a1546b-698d-4ed0-8b38-69bdfc2300e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:54.664334 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a1546b-698d-4ed0-8b38-69bdfc2300e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:54.665988 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a1546b-698d-4ed0-8b38-69bdfc2300e3 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:54.676505 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1468/2928] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 10:02:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 10:02:54.681061 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5df758-a267-4d97-8f5a-a6e910403ad0 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:54.690873 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5df758-a267-4d97-8f5a-a6e910403ad0 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:54.695439 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a0ff7d-1d2d-4818-af9f-9d4eb6a8b301 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:54.696024 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a0ff7d-1d2d-4818-af9f-9d4eb6a8b301 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:54.696301 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a0ff7d-1d2d-4818-af9f-9d4eb6a8b301 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:54.696573 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a0ff7d-1d2d-4818-af9f-9d4eb6a8b301 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:54.697200 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a0ff7d-1d2d-4818-af9f-9d4eb6a8b301 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:54.698822 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a0ff7d-1d2d-4818-af9f-9d4eb6a8b301 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:54.709326 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1461/2929] 10.222.0.39 () {58 vars in 1122 bytes} [Fri Sep 30 10:02:54 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 30 10:02:54.713163 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a1546b-698d-4ed0-8b38-69bdfc2300e3 admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:54.723119 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a1546b-698d-4ed0-8b38-69bdfc2300e3 admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:54.728380 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b508ab-7677-4565-ac28-a8223a102feb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:54.729048 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b508ab-7677-4565-ac28-a8223a102feb admin admin] REQUEST_METHOD: `PUT` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:54.729352 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b508ab-7677-4565-ac28-a8223a102feb admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:54.729623 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b508ab-7677-4565-ac28-a8223a102feb admin admin] PATH_INFO: `/v3/projects/6da128f02e0549a499923e6db6b5651d/users/302d5c84218341b097acb9ca62c42ccd/roles/025f58735650469d9fa6faca3e9e7998` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:54.744413 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b508ab-7677-4565-ac28-a8223a102feb admin admin] RBAC: Authorizing `identity:create_grant(project_id=6da128f02e0549a499923e6db6b5651d, user_id=302d5c84218341b097acb9ca62c42ccd, role_id=025f58735650469d9fa6faca3e9e7998)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:54.747088 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b508ab-7677-4565-ac28-a8223a102feb admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:54.819667 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1469/2930] 10.222.0.39 () {60 vars in 1483 bytes} [Fri Sep 30 10:02:54 2022] PUT /identity/v3/projects/6da128f02e0549a499923e6db6b5651d/users/302d5c84218341b097acb9ca62c42ccd/roles/025f58735650469d9fa6faca3e9e7998 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 30 10:02:54.830889 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe3301c-556d-476d-90aa-27f18cc67257 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:54.831174 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe3301c-556d-476d-90aa-27f18cc67257 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:54.831444 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe3301c-556d-476d-90aa-27f18cc67257 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:54.859271 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-cfe3301c-556d-476d-90aa-27f18cc67257 None None] MFA Rules not processed for user `302d5c84218341b097acb9ca62c42ccd`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:02:54.860517 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-cfe3301c-556d-476d-90aa-27f18cc67257 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:54.923973 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1462/2931] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:02:54 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:02:54.936107 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b508ab-7677-4565-ac28-a8223a102feb admin admin] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:02:54.937814 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-97b508ab-7677-4565-ac28-a8223a102feb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:02:54.959509 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:02:54.996417 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b508ab-7677-4565-ac28-a8223a102feb admin admin] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:02:55.000154 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cedbc9aa-bf51-42d2-93e2-3bf7ff7be955 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4e2742f567444dff88ed919be52217de', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:02:55.001245 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedbc9aa-bf51-42d2-93e2-3bf7ff7be955 service cinder] REQUEST_METHOD: `GET` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:02:55.001478 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedbc9aa-bf51-42d2-93e2-3bf7ff7be955 service cinder] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:02:55.001759 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedbc9aa-bf51-42d2-93e2-3bf7ff7be955 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:02:55.010665 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cedbc9aa-bf51-42d2-93e2-3bf7ff7be955 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:02:55.012144 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cedbc9aa-bf51-42d2-93e2-3bf7ff7be955 service cinder] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:02:55.021486 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1470/2932] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 10:02:54 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 10:03:04.921092 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:03:05.100336 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=17 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:03:06.497519 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe3301c-556d-476d-90aa-27f18cc67257 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:03:06.560451 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe3301c-556d-476d-90aa-27f18cc67257 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:03:06.564842 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32bdb12-f135-49a3-8544-f9f7586ce6f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:03:06.565459 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32bdb12-f135-49a3-8544-f9f7586ce6f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:03:06.565738 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32bdb12-f135-49a3-8544-f9f7586ce6f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:03:06.566008 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32bdb12-f135-49a3-8544-f9f7586ce6f6 admin admin] PATH_INFO: `/v3/users/302d5c84218341b097acb9ca62c42ccd` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:03:06.567132 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32bdb12-f135-49a3-8544-f9f7586ce6f6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=302d5c84218341b097acb9ca62c42ccd)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:03:06.569033 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32bdb12-f135-49a3-8544-f9f7586ce6f6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:03:06.803819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d32bdb12-f135-49a3-8544-f9f7586ce6f6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '302d5c84218341b097acb9ca62c42ccd'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:03:06.804474 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d32bdb12-f135-49a3-8544-f9f7586ce6f6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '302d5c84218341b097acb9ca62c42ccd'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:03:06.805073 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d32bdb12-f135-49a3-8544-f9f7586ce6f6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '302d5c84218341b097acb9ca62c42ccd'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:03:06.812180 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d32bdb12-f135-49a3-8544-f9f7586ce6f6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '302d5c84218341b097acb9ca62c42ccd'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:03:06.875845 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d32bdb12-f135-49a3-8544-f9f7586ce6f6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '302d5c84218341b097acb9ca62c42ccd'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:03:06.894421 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1463/2933] 10.222.0.39 () {58 vars in 1224 bytes} [Fri Sep 30 10:03:06 2022] DELETE /identity/v3/users/302d5c84218341b097acb9ca62c42ccd => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:03:06.899489 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d49564-48b2-4778-9b0a-a658a887b1f5 None None] REQUEST_METHOD: `POST` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:03:06.899972 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d49564-48b2-4778-9b0a-a658a887b1f5 None None] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:03:06.900080 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d49564-48b2-4778-9b0a-a658a887b1f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:03:06.919436 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.auth.core [None req-07d49564-48b2-4778-9b0a-a658a887b1f5 None None] MFA Rules not processed for user `767ee8fd8728440799a75a0b00b134c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81685) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 30 10:03:06.920913 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-07d49564-48b2-4778-9b0a-a658a887b1f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:03:06.978277 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1471/2934] 10.222.0.39 () {58 vars in 958 bytes} [Fri Sep 30 10:03:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 30 10:03:06.987812 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32bdb12-f135-49a3-8544-f9f7586ce6f6 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:03:06.989256 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d32bdb12-f135-49a3-8544-f9f7586ce6f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:03:07.045094 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32bdb12-f135-49a3-8544-f9f7586ce6f6 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:03:07.048605 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2e31b7-a8ae-4dd6-96e2-09fb5b833300 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9266babd21414ae9998716e5a0478e9d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:03:07.049199 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2e31b7-a8ae-4dd6-96e2-09fb5b833300 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:03:07.049475 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2e31b7-a8ae-4dd6-96e2-09fb5b833300 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:03:07.049745 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2e31b7-a8ae-4dd6-96e2-09fb5b833300 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:03:07.058894 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2e31b7-a8ae-4dd6-96e2-09fb5b833300 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:03:07.060298 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2e31b7-a8ae-4dd6-96e2-09fb5b833300 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:03:07.070146 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1464/2935] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 10:03:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 10:03:07.428480 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d49564-48b2-4778-9b0a-a658a887b1f5 None None] Authenticating user token {{(pid=81685) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:03:07.429982 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.fernet_utils [None req-07d49564-48b2-4778-9b0a-a658a887b1f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81685) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:03:07.442763 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d49564-48b2-4778-9b0a-a658a887b1f5 None None] Validating token access rules against request {{(pid=81685) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:03:07.447481 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381ec329-d91b-4c14-b067-2d5f7d126ce9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '767ee8fd8728440799a75a0b00b134c0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f04f50d086c64df3bc18b4c8f8baadc4', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81685) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:03:07.448123 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381ec329-d91b-4c14-b067-2d5f7d126ce9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:03:07.448405 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381ec329-d91b-4c14-b067-2d5f7d126ce9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:03:07.448682 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381ec329-d91b-4c14-b067-2d5f7d126ce9 admin admin] PATH_INFO: `/v3/projects/6da128f02e0549a499923e6db6b5651d` {{(pid=81685) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:03:07.449895 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381ec329-d91b-4c14-b067-2d5f7d126ce9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6da128f02e0549a499923e6db6b5651d)` {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:03:07.451779 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381ec329-d91b-4c14-b067-2d5f7d126ce9 admin admin] RBAC: Authorization granted {{(pid=81685) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:03:07.541084 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-381ec329-d91b-4c14-b067-2d5f7d126ce9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6da128f02e0549a499923e6db6b5651d', 'request_id': 'req-381ec329-d91b-4c14-b067-2d5f7d126ce9'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:03:07.546805 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-381ec329-d91b-4c14-b067-2d5f7d126ce9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6da128f02e0549a499923e6db6b5651d', 'request_id': 'req-381ec329-d91b-4c14-b067-2d5f7d126ce9'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:03:07.553120 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-381ec329-d91b-4c14-b067-2d5f7d126ce9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6da128f02e0549a499923e6db6b5651d', 'request_id': 'req-381ec329-d91b-4c14-b067-2d5f7d126ce9'} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:03:07.647896 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-381ec329-d91b-4c14-b067-2d5f7d126ce9 admin admin] The token cache is being invalidate because project 6da128f02e0549a499923e6db6b5651d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81685) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 30 10:03:07.648124 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-381ec329-d91b-4c14-b067-2d5f7d126ce9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:03:07.648758 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG keystone.notifications [None req-381ec329-d91b-4c14-b067-2d5f7d126ce9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81685) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 30 10:03:07.650242 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: [pid: 81685|app: 0|req: 1472/2936] 10.222.0.39 () {58 vars in 1233 bytes} [Fri Sep 30 10:03:07 2022] DELETE /identity/v3/projects/6da128f02e0549a499923e6db6b5651d => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 30 10:03:17.065460 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=79,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:03:17.644132 dv-cinder-master-3-iscsi devstack@keystone.service[81685]: DEBUG dbcounter [-] [81685] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81685) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 30 10:05:13.576873 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2e31b7-a8ae-4dd6-96e2-09fb5b833300 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 30 10:05:13.578282 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fc2e31b7-a8ae-4dd6-96e2-09fb5b833300 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:05:13.714862 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2e31b7-a8ae-4dd6-96e2-09fb5b833300 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 30 10:05:13.772819 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be36934-7380-4dfa-9b5b-ac67b2833a51 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '49a3011c103b4e48904f876f81d27fba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c3802ced1fce4997a016d41eca843ef8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 30 10:05:13.773416 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be36934-7380-4dfa-9b5b-ac67b2833a51 service placement] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 30 10:05:13.773699 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be36934-7380-4dfa-9b5b-ac67b2833a51 service placement] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 30 10:05:13.773974 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be36934-7380-4dfa-9b5b-ac67b2833a51 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 30 10:05:13.775724 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5be36934-7380-4dfa-9b5b-ac67b2833a51 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 30 10:05:13.786790 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be36934-7380-4dfa-9b5b-ac67b2833a51 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 30 10:05:13.788990 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be36934-7380-4dfa-9b5b-ac67b2833a51 service placement] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 30 10:05:13.798677 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1465/2937] 10.222.0.39 () {60 vars in 1365 bytes} [Fri Sep 30 10:05:13 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 30 10:05:23.794519 dv-cinder-master-3-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}